My question is: If what Grog is doing is so good, why can't Nvidia copy it?
As far as I know, Grog AI accelerators are really fast because they load the model into SRAM only and spread the model out over many Grog chips. They don't use HBM or off-chip RAM at all.
That doesn't seem like a defensible moat to me.
Right now, I believe Nvidia's GPUs are geared more towards training. These startups tend to compete more in inference since it's simpler. I do think Nvidia needs to shore up their inference offerings more. Actually, buying Grog or starting their own inference-only chips aren't bad ideas.
It takes a while to design, manufacture, and test/validate new silicon. There is also a very elaborate software stack on top of it. This is a multi-year effort to reproduce for an extremely talented, nimble engineering team. And by then, Groq will have even more refined and powerful systems available.
Also, when an end market is growing as fast as the AI compute market is, multiple players can do very well for years at a time. I think Groq also has a high probability of being acquired by Google/Microsoft/Apple/Meta if the FTC allows it. But they are probably better off following the hockey stick growth for a couple more years before doing that.
Nvidia should have a far more well oiled machine in designing/validating new silicon. Software stack? Nvidia already has that. All models are optimized for CUDA - not Grog.
At a very high level, Nvidia just needs to break out the Tensor cores into its own chips, add loads of SRAM, and run CUDA. They already have all the other datacenter stuff figured out such as interconnects.
As far as I know, Grog AI accelerators are really fast because they load the model into SRAM only and spread the model out over many Grog chips. They don't use HBM or off-chip RAM at all.
That doesn't seem like a defensible moat to me.
Right now, I believe Nvidia's GPUs are geared more towards training. These startups tend to compete more in inference since it's simpler. I do think Nvidia needs to shore up their inference offerings more. Actually, buying Grog or starting their own inference-only chips aren't bad ideas.