Should it be illegal to have sex with someone other than your spouse?
In some places, it is. But keep in mind that, as you say, a marriage is a contract. Violations of a contract are matters of civil law, not criminal law. The police dont come and arrest someone for violating a contract; instead, the aggrieved person sues for redress. For example, a divorce. Also, keep in mind that not all contracts are the same. I am married. My wife and I did not pledge sexual exclusivity. Do you think the government should spend money and resources investigating us for having sex with other people, given that our contract doesnt prohibit it?