> key auth means the machine is authorized on your server

Not necessarily: Depends on whether your key is passphrase-protected and how your SSH agent is configured (if you use one). You can have the standard OpenSSH one ask you for confirmation of every key usage, for example.

> consider a vpn please

But also consider how you'll fix a broken VPN without SSH access.