To go on or not to go on (with the series), that is the question!

I'm currently rereading "Outlander" the first book in Diana Gabaldon's series (of the same title). I loved it the first time I read it and I'm still loving it now. It's not an easy book to read, but it's powerful and emotional at the same time and I just adore Jamie and Claire and the Happy End they got in "Outlander". This was also the reason why I didn't go on with the series. I decided to treat the first book like a standalone and avoid all the following drama.

But now my mom - who loves the series, but refuses to reveal too much, hmph >:/ - is telling me to at least read the next three books "Dragonfly In Amber", "Voyager" and "Drums Of Autumn". I'm just not sure if I should. I'm afraid that at some point I will stumble over something that will ruin the love story for me. The one thing that I can't forgive a character for is cheating. I insist on fidelity and some of the rumours I heard worried me quite a bit.

If you haven't read the series - spoilers ahead! 

I'm aware of the fact that Claire and Jamie  are apart for twenty years and I don't hold their relationships/affairs during that time against them - they aren't saints and twenty years is an awfully long time to spend alone. But if Claire and Jamie cheated on each other once they found each other again, it would ruin the whole series for me. I don't care if this series takes place in another time, with different customs and morals.

As you can see I'm in a bit of a bind. On the one hand I'm curious about Jamie and Claire's journey, seeing how so many people love this series. But at the other hand I'm worried about ruining a great book like "Outlander" by going on with a series that's just not for me.

So did you read the series? Would you advise me to go on with the series or just let it be and treat "Outlander" as a standalone novel? What were your general thoughts on this series? Feel free to include as many spoilers as you want. I'd be very grateful for your help. All this wondering is starting to drive me crazy!

Recpectively, have you ever had a hard time deciding if you want to go on with a series? What were your decisive factors?