> This can have a lot of knock on effects, like leaving early (really, leaving before others) from work is inconveniencing your peers and your boss.

Does this go the other way too? Is the boss staying late (with the expectation that others do so as well) considered rude because it inconveniences his workers?

Because if not, this has nothing to do with being polite and everything to do with power and deference to power.