Should I go swimming with this guy I barely know?

No. Why even consider it. It's clearly something you do not want to do! So speak up and politely say you are not going.