What Ben Affleck was spotted doing on vacation with his new girlfriend has left some fans questioning him