An LLM that could mutate its own weights might be closer to it. The various labs must have experimented with this, but it seems that it would be economically prohibitive to productize.
That's my ignorant thesis since the launch of GPT 4. Everytime I mention it, someone says it is impossible. But most AI labs are working on big models that serves lots of people simultaneously so the shared context wouldn't make much sense. I think we will see more on this now that local models are gaining more traction and capturing the hype.
