Most DJs do not make their livings at techno clubs. The majority are hired to play bars and events that do not cater to particularly discerning audiences.
Exactly. The DJs that are innovative, crate digging, slightly pretentious music nerds are almost exclusively hobbyists, with a vanishingly tiny percentage of them being able to eek out a meagre living from it. The majority of full time DJs cater to mainstream audiences who absolutely do want to hear the same 50 - 100 tracks on rotation every time they go out. They want to dance and sing along to music that they're familiar with, and if the DJ doesn't play what they know then they won't dance, they won't stay, and the DJ won't remain employed.
It's actually very similar to web design - innovation has its place, but 99% of the time people want familiarity.
A typical DJ is allowed one weird song nobody has heard before. If it is a long show maybe one per hour. The rest better be songs the majority of people know and sign along with.
You might have a bit of confirmation bias based on the particular environments you've been in.
Having taken part in various types of electronic music/art scenes since the early 2000's, I've met all kinds of people. Local hobbyist bedroom producers playing for free. Semi-professional artists juggling gigs&touring with one or more side jobs. Full-time DJ's playing everything from small underground parties to some of the biggest parties/festivals at the time. They all cater to their audience to varying degrees, mainstream or not.
Granted, the scenes I've bumped into tend to be on the non-mainstream side. That's where you can actually go professional being that "innovative, crate digging music nerd" you refer to (removed "slightly pretentious" because that hasn't been my experience). It's tough, but it can be done, and it's a larger group of people than you seem to think.
I've also met some professional DJ's that fully cater to the audience in the way you describe. Many of them make statements like yours like e.g. "99% of the time", "almost exclusively hobbyists", "slightly pretentious", etc. I really don't get why, because it's just not true, and it comes across as a bit defensive or passive-aggressive to be honest.
I mean, of course there is the mainstream audience of the type you describe. But even that audience changes its opinion about which 50-100 tracks they expect you to play on a regular basis. That change has to come from somewhere, otherwise they'd still demand disco tracks from the 70's. That somewhere is the stuff that hasn't gone mainstream yet, and while the percentage of people that can make a living of it is probably not very high, it's a lot higher than what you claim it to be.
That vast overlap between underground/alternative scenes and the mainstream is super interesting, and I'm pretty sure that if you included that part into your statistics, you'd see a different picture.
NB: I might have a bit of confirmation bias based on the particular environments I've been in ;)
