I think certainly directing is a visual medium, but it's also about communication, and a lot of times great directors are lacking in communication skills, which is rather shocking to discover that.
When you work with your wife you really have no "me" time, or place to get away and be with the guys, because you're at work all day together, and then you're at home, and you have this other public life.