How about "we are taxing your wealth because the money has to come from somewhere, and we can't take it from people who don't have any"? Given the diminishing marginal utility of income and wealth[0] it makes economic sense to tax the people who have a lot of wealth at a greater rate than those who have a little.
To try to fill in the blanks and steelman your position a little, I think that the most significant difference (and potential grounds for criticism) is that a wealth tax means the same money is taxed multiple times.
I'm not sure what essential philosophical or ethical objection there is to that, though, since money is constantly changing hands and being taxed (e.g. on the way in as income, and on the way out as consumption/sales tax). Moreover, as taxation changes incentives, it should be targeted to reduce socially negative actions, and arguably "wealth hoarding" is worse for society than "income", which is currently an accepted target for taxation.
I think the biggest "fairness" argument for a wealth tax, though, is that the average US household has a net worth of $120k, and pays $10k in total taxes per year, which is equivalent to an 8% wealth tax. (Of course it's true that if the household decided to not earn any money, it could greatly reduce its tax burden, but that's not really an option for most tax payers, whereas billionaires never have to work a day in their life).
So I would tie the wealth tax to be equivalent to the effective rate that the median household pays, but allow deductions for any other payments the wealthy person paid to the government or to charities. Also, for simplicity, the wealth tax should only apply to households above a certain wealth threshold, perhaps 10x or 100x of the median.
[0] https://www.economicshelp.org/blog/12309/concepts/diminishin...