Not only are those numbers not always right (they can contain replies), but also that's not the question. If the questions are about keyword presence, it's easier but it depends a lot on stuff like, "do we count attachment content?"
I have to agree with the other commenter, I think you're definitely overthinking it. A keyword search is clearly what they meant for the user to do.
They're not expecting 5% of the population to train a word2vec model and produce prediction interval estimates. Just a simple back-of-the-envelope check.
Maybe they should word the question that way then? No one is asking for using widely criticized ML tooling.
The addition of the word "estimate" doesn't break the word budget and opens a lot of flexibility when it comes to unfamiliar tooling. Even if we ignore that, I reiterate: who has ever asked that question? The outlandishness of that question surely has at least as much responsibility as the obscurity of the UX in question or the general population's lack of education.
Personally I think most of the actual tasks have so little bearing on the reality of the average computer user's needs I can't say I'm surprised folks couldn't succeed. I also think this is a cognitive trap for a lot of folks here used to being skilled computer users; becoming eager to pontificate on why our unique insight is what's needed to "fix" the problem.
I just think that you're focusing on the wrong things here. Towards the top of the article they say:
> One of the difficult tasks was to schedule a meeting room in a scheduling application, using information contained in several email messages. This was difficult because the problem statement was implicit and involved multiple steps and multiple constraints. It would have been much easier to solve the explicitly stated problem of booking room A for Wednesday at 3pm, but having to determine the ultimate need based on piecing together many pieces of info from across separate applications made this a difficult job for many users.
This task isn't overly complicated. It's a task that computer-savvy people do on a regular basis, but it complex enough that it might trip up someone who can't use a computer well.
It's also a task people screw up routinely. Reliably getting a room for a meeting in a busy building is not a solvable problem. And again, it's a problem the vast majority of computer users simply do not and will never have.
The self-similar biasing in this study us almost ridiculous.
