'But when people talk about "mental maps," I'm not entirely sure what they're talking about. '

For me, knowing where things are in the space around me or in the larger navigation zone is like knowing where the parts of my body are. After spending just a little time in a space, it all feels like one thing and I can mentally point to various places within that space.

I can still get turned around if I have been going through multiple corridors inside with no visibility outside, particularly underground and particularly if some of the corridors are slightly off of rectilinear. Then I can come out thinking I am facing south and it turns out to be east. However, the next time I move though that space, chances are it will all connect up in my mind and then I will know where I am and which direction is which.

My husband is a little more like you. When we exit a building, he will seemingly choose a direction at random. It almost seems like he kind of gets it but backwards and almost always goes the opposite direction. He is not as far off as you. He can learn to navigate familiar spaces but it takes him a long time.