I'm sure George explained it better, but I'll try.

Between-landlord competition will keep building prices at a level that's equal to the value they create, but it does nothing to ensure that the surplus gets fairly distributed. If you imagine a town of millers and bakers, as long as people can freely switch jobs then both will be equally profitable careers, because they'll have an equal share of the surplus. But if you only allow certain people to be millers then the price of flour will rise to the point where a baker can only just get by, while the millers will grow rich because they capture all of the surplus. There's no incentive for a landlord to sell at any less than the full value that their buildings are worth - and eventually the full value of all economic activity from people living in their buildings.

You're not defining what you think "fairly distributed" means, or why a landlord shouldn't benefit fully from a rise in real estate prices, any different from how a stockholder should benefit fully from a rise in stock prices.

Remember, owners are investors. Landlords gain when economic growth happens, the same way they lose when the economy shrinks. They put up capital and pay interest to invest in buildings they think will gain in value. And as prices go up, landlords pay higher property taxes too (and unlike capital gains, they can't wait until they sell).

But landlords aren't generally capturing the "full value of all economic activity" from people living in their buildings! That doesn't make any sense -- as people's salaries go up, they want nicer buildings/neighborhoods, but they also want nicer vacations, nicer food, nicer schooling for their kids, and so on. Landlords capture some of the economic growth of a region, but only to the extent that's what tenants prefer to spend their income on. The same way Starbucks captures growth, but only to the extent people want to spend their money on nicer coffee -- no more, no less.

The NIMBY/zoning conversation is a real one... and it's up to a community to decide democratically if they prefer less housing that is more expensive, or more construction for more affordable rents. But I don't see how there's anything more or less "fair" about one side or the other. It's a question of what the community wants. Manhattan would have cheaper housing if it replaced Central Park with blocks of high-rise buildings. That isn't a question about fairness, though.

There's no absolute "should" for what you're talking about, but the argument most economists would make is that most other tax schemes other than land value taxes are economically inefficient. One way this can manifest is deadweight loss.

This means that we misallocate resources.

For instance, consider a man who buys a lot of land next to city hall. He does nothing with it for years as the city grows up around him. His lot is now worth a lot of money and he pays very little for it yearly. One way of looking at this is in terms of how much it costs to provide him services: nothing, so his property tax is net positive. Another way is that there is a net loss to society because his property is not put to use in an economically efficient way.

Economically efficient does not necessarily mean developed, by the way. Golden Gate Park may be an economically efficient allocation of resources, though it is not used for built up space.