I wonder if that could be solved with a similar sort of design. Instead of a lock with a master key, how about a "break in case of fire" cover on the elevators? You could still have a lock and building-specific keys for non-emergency uses.

Targets for vandalism and other shenaniganery.

How often have you heard of someone picking the lock to a knox box or an elevator control panel? (Probably not very)

How often have you heard of someone hitting the alarm or screwing around with the emergency options? (Probably more)

Well, unlike standard fire alarms, these don't need to be easy to break into. Since they're for use by the fire department in an emergency, you can assume that whoever uses them will have access to axes and a willingness to use them.