I guess a lot of those people end up in poverty, though probably not as many as it would seem it should, because the society does a lot of work to protect unwillingly innumerate people from exploitation, which makes it easier for lazy people to disregard numeracy without consequences.

Still, numeracy is not being able to multiply a bunch of 3-digit numbers on a piece of paper. Numeracy is primarily knowing which numbers to multiply, or add together, or subtract from each other to learn something you need, and that you can do this in the first place.