That's initially true, but as the car ages, and maintenance is done (or not done, or done improperly) the car's aspects will change.

And as new cars are developed, obviously there are more "drivers". Essentially every year there'll be new drivers for every company that makes autonomous cars, possibly multiple.

Yeah, it's still a lot fewer than the individual humans, but it's definitely not just "one driver".