> what is something that only an actually thinking entity can do that LLMs can't?

This is pretty much exactly what https://arcprize.org/arc-agi is working on.