Steve Harvey has the only legitimate reason for being the latest to announce he will not attend the inauguration