To support your theory, I am an extreme introvert managing a fully distributed team. Yet I choose to regularly come to the office for a bit of in-person interaction (mainly social since my direct co-workers aren't based here).

I should mention that I live within a ten-minute walk to the office. For example, I can easily walk there for lunch and then walk back home. If the commute was much longer, that would significantly change the calculus. A pleasant half-hour train ride? Maybe, but not very frequently. An hour drive? No way I'd go unless I absolutely have to.