It’s not just about being able to live on savings. When you come from a poorer community you can’t really afford to take on an expensive education and mediocre pay, because the outcome of that expensive education generally requires you also support your parents/family that helped you get there.
But moreover yeah, richer families can do more outdoorish activities like mountaineering and that feeds back into the loop of even being interested in something like this.
Many, and in America I'd dare say most, people just don't really like doing outdoor things. It's some of the cheapest entertainment you can have, as many parks and locations are completely free. You don't need to full on belaying to have a great time. All you really need is a tank of gas, a bus ticket, or a bike. There are just dramatic cultural differences that drive people in different directions.
Programs like ROTC/JROTC also offer countless outdoor opportunities alongside other possibilities for personal enrichment. I'm vehemently antiwar/hegemony, but I think training offered by ROTC/JROTC is extremely valuable for anybody and everybody. It's also completely free.
But moreover yeah, richer families can do more outdoorish activities like mountaineering and that feeds back into the loop of even being interested in something like this.