Why do we care so much about infidelity?

My wife hasn't cheated on me and nor have i, but why am i so afraid of it happening. Every time we argue thats the first conclusion i come to "YOU MUST BE SLEEPING WITH SOMEONE ELSE" it's getting really tiresome. plz no answers.