Or just tax them differently than a primary residence.

Stable and long term living arrangements are something the government should be incentivizing so primary residences should be taxed minimally.

Vacation houses and income properties should be taxed higher.