Not that I agree but Governments take a very nuanced view on what they consider impact. Remember, growing your own corn for feed outside of the government quota was considered as having an impact on "Interstate Commerce".

In a hypothetical; If you were employed by an Canadian company, doing internal tech support and then you then move to Belize because you hate the cold. No interaction with anything in Belize.

The Belize government might say, "Not so, you moved here and are taking advantage of our infrastructure, societal governance, etc. but are not helping pay for it." Furthermore, natives of Belize may be upset that because you have a higher income than the native population you are affecting the cost of goods(not one person individually but if 1000 people with high income show up and are all looking to rent houses, the rental rates would rise both due to demand and the ability to pay.)

If we think about things especially from the second point of view, it would be very bad for a country to allow unlimited rich foreigners to come into their country and live there; Even if they are contributing by purchasing local goods and services, they are in competition with local workers as well; They might even stunt the economy if it becomes reliant on them and then they all move out suddenly; and the foreigners would also start shaping the culture of the country as well, which could be neocolonialism depending on your view of things.

EDIT to add: I like the idea of being able to live wherever I want to but it is also important to consider the views of those who are in those countries as well. I think that some(most?) people from Europe and North America don't have a strong national identity and don't see any value in such things. But there are people in other countries that do take a lot of pride in where they are from and how long their family has lived there. Some of this pride is what builds the culture and the local charm that those without enjoy but it becomes a sort of "We are destroying the thing we love by trying to experience it" situation.