> And the user has to trust that these 'knowledgeable people' are indeed knowledgeable

Actually, the user doesn't have to trust anyone: the user can ask more than one party to audit the code. In fact, with open source, there are, usually, many different parties already auditing the code.