At least 4 U.S. Olympians have already said they wouldn’t accept an invite to the White House from Trump