Passengers can just grab the phone and do whatever they want.

Except apps like Waze will show you "this app is being controlled by Android Auto" and won't let you do anything on the phone's screen, you have to use the Android Auto display to interact with it.

Which is something the driver can do too, which is why this stupid restriction is making the system less safe.

If they do, it’s not on Apple or the car manufacturer for making it unsafe. There are a laws all over the world about having an infotainment system and distractions.