I think so. Mostly:

- Hosted solutions charge you for tokens. More tokens, more money. Keeping money in your pocket: generally recognized as rad.

- 1M tokens wouldn't hold the entire codebase I have open in my other window.

I'm still not sold on recall at such large context window sizes. It's easy for an LLM to find a needle in a haystack, but in most RAG use-cases it's like finding a needle in a stack of needles, and the benchmarks don't really reflect that. There's also the speed and cost implications of dumping millions of tokens into a prompt - it's prohibitively slow and expensive right now.

It's still much cheaper to run RAG in production (at least if you are using closed models). I'd love to use the entire context of GPT4, but if I do that in production it'll cost much more than using some RAG-dependent implementation.

But this is just current state. Token costs continue to go down and contexts will continue to get larger.

The LLM is always out of date. RAG is here to stay.

correct me if i'm wrong but you still have to get the context populated. It's still the RAG pattern you just put more data in the prompt than before.

Well I guess the assumption is that you can just fit everything into the context and there'd be nothing to retrieve anymore.

yes - private data, real-time data, curated data, citations with no hallucinations, RAG on tabular data, RAG on video, RAG on hierarchical mixed data, RAG over a graph

Memory-constrained environments.

If you're running a 1M context LLM I assume you are well past memory constraints since the weights alone are 10s of GBs.