The comedian and actress claimed she and her husband were labeled as ‘difficult’ for saying ‘no.’