I think the hallucinations show that it's not simply overfit to all of human knowledge. To hallucinate, there is a certain amount of generalization and information overlap that is necessary.