Better to ignore public servants' private lives and just grade them on their job performance. Keeps you from losing people who are doing their work properly.
Update: Many right-wingers are seeing all this as part of a Benghazi-related conspiracy. Foreign Policy considers several of the theories offered by people who can't spell Petraeus. I understand leaving out the 'e', but adding an 'o' or a 'y' is a larger mistake.