Are the same embeddings not used internally? I thought they were. Maybe I'm wrong about that.
Mistral uses a 1024 dimension embedding for 8K context. I think the point about trying to capture that rich of a context into a smaller number of dimensions still stands?
For long contexts this is a key consideration along with what self attention optimizations the model chooses to implement.
They don’t make this public, but we can infer they can’t be using full self attention pairs at 1,000,000 tokens because it scales quadratically and would take Terabytes of RAM.
There are different approaches like sparse attention, and the only way to really know how well their choices work is to test it.
Mistral uses a 1024 dimension embedding for 8K context. I think the point about trying to capture that rich of a context into a smaller number of dimensions still stands?