I really have to wonder if anyone can compete with this kind of systems integration capability. A core having 900GBps connectivity to the cluster memory at such relative low power is epic beyond words. 800Gbps ethernet across PCIe is uncompetitive in extreme.
How the rest of the industry can respond is such a mystery. And will it be lone competitors, or will a new PC era be able to start, with an ecosystem of capabilities?
This seems to be competing directly with Google's TPU pods. Looks like TPU v4 has a 300 GB/s interconnect, and 32 GB HBM per chip * 4096 chips = 131 TB (which is all HBM, so higher bandwidth than the LPDDR in Nvidia's system). So yeah, Nvidia's interconnect seems better. However, TPU v4 was deployed in 2020 (!) and Nvidia's thing won't be ready until next year. I've gotta imagine that TPU v5 has already been deployed internally for a while now, but hasn't been disclosed yet. Who knows, TPU v6 might even be deployed before this Nvidia thing.
Just want to flag a potential unit issue: 900gbps vs 300GB/s?
Also worth noting - TPUv4 uses a 6-way 3D torus interconnect vs the 3-way "multi ToR" NVLINK topology; the total bisection bandwidth of the TPUv4 pod is over 1PB/s!
Can't wait to see what TPUv5 looks like. As you say, it's probably already chugging away with v6 on track to tape out in a year.
That said, I think NVidia has nailed bringing the ecosystem along, and I think making the whole setup look more like "one huge GPU" could simplify a lot of ML programming.
I am actually disappointed I haven't seen more of that style in CPU programming. Where's my 20,000 core 100TB RAM VM instance?
The nvidia device uses a fabric with 900 GBps switched fabric between any of the 256 nodes in the system. The TPUv4 3d torus network is basically a ring network of 56 GBps connections creating separate rings. From a raw perspective, the nvidia solution is the overwhelming winner. There is absolutely no contest.
You could simulate this with a bunch of regular machines and a networked hypervisor.
You could do some kind of smart caching so that processes rarely need to wait to access RAM stored on a remote machine.
Combined that with a big lock eliding/speculation scheme (ie. When a process reads memory that might have been written by a remote CPU, you continue as if it hadn't, and if you later find out that data was written then you rollback). These rollbacks 'undo' all work done in however many microseconds it takes for data to travel from one side of the machine cluster to another.
Reads of RAM that aren't cached yet on the local node can also be speculated - you just assume that RAM contained null bytes and continue execution, rolling back and replaying when the actual data arrives.
So if you can make sure that processes are contending for locks and writing conflicting data less often than once per system-roundtrip-latency, then you should get a high performance system.
This is certainly a very interesting thought to entertain and your ideas make sense. One thing that makes things harder on the CPU side in this hypothetical scenario is that CPUs tend to execute much more diverse instructions/computations than GPUs. So all the caching & speculation you mention is probably all the more important.
After writing the comment, I considered writing a little toy example just to try out the idea... It would be neat to see Linux boot with 1000 CPU's...
But upon further thought, a lot of things such a system would need are actually rather inefficient to implement in software (ie. rollbackable RAM), yet quite cheap in hardware (for example rollbackable RAM can be implemented with regular RAM plus either a buffer of 'overwritten data' or a write queue)
Machines with terabytes of RAM do exist and get used - working well on such setups is a goal of modern JVM GCs for instance - but making a single machine that large which acts like a single machine isn't easy, nor especially desirable. One machine is a unified failure domains outside of mainframe-land, so if you had a 20k core machine with 100TB of RAM you could never reboot it to apply OS updates and it'd die all the time from failed parts.
Even once you get beyond that most software stacks use locking and stop scaling beyond a few hundred cores at best and that's assuming very heavily optimized stacks. AI workloads are easier because they're designed from scratch to be inherently parallel without lots of little locks and custom data structures all over the place like a regular computer has.
Disk storage is one of the places where you can parallelize and scale out relatively easily and you do see datacenter sized disks there.
Software for TPU is still in its early stages. CUDA is well established. You can test on a gaming GPU that you can find (locally!) in many markets. XLA is meant to solve this, but first impressions matter and my first impression was that it has not yet "solved" this issue.
TPU is only available via Google Cloud - as far as I know they don't have NVIDIA's widespread distribution to various HPC/supercomputer systems. This also has implications on scaling up more than a few pods, as they will need to be colocated with speedy interconnect (which is provided by the various existing HPC systems that use NVIDIA's chips).
Finally, I think many people are discovering that the supposed benefits of TPU are marginal at best in the face of the types of natural scaling issues that both GPU's and TPU's suffer from when scaling out to e.g. hundreds of pods.
I'm certain that someone with more experience than I could give a better answer though - and again, all speculation. I refuse to use TPU because Google Cloud's system for getting access to said TPU's was horrible for me when I tried it. I believe John Carmack has a nice tweet thread specifying the same issues I ran into.
In general, Google has a habit of developing tech for other Googlers first, and as such winds up ignoring a lot of real-world scenarios faced by researchers/practitioners. NVIDIA on the other hand has been working directly with a ton of institutions and businesses ever since the inception of CUDA.
That their TPU's have seen any adoption at all is mostly due to their research program which granted very cheap access to TPU's to tons of people.
This is starting to sound very paperclippy. Ads fund the AIs to make us click on ads to fund AIs that are even better at getting us to click on even more ads.
Yeah, and look at how some very simple clustering ML/recommender systems impact social/political dynamics all to keep people engaged on the site and maximize chances to click ads ( see youtube/facebook, etc. ).
Damn right but I don't understand why. That is, why is ads business generating so much profits that it allows to build such ridiculously powerfull devices ? Is it because it's genuinely full of money or is it because Google is so central that it makes tons of money out of lots an dlots and lots of small adverts ?
It's a monopoly on eye balls. People don't casually walk in front of domain names, they must find them on Google
As a result, spending ad money on Google is ridiculously expensive, but companies accept this because there is no alternative hoping to "build long lasting relations" with the people who make them pay upwards of 1 dollar per click
At the same time it is also a huge bubble, that Google is just hoping will never burst. People and businesses way overestimate the impact their ads are having and way underestimate the impact, that treating customers well can have.
I definitely think this is the strategy of google leaders, they've heard to much of "how do you monetize your products?" from investors and now they are maximizing profits for that current software generation. I wonder though if that bulk of money will be that much of an advantage when the tides turn. It could attract the wrong kind of leadership amongst other things like customer distrusts and turn the company into an IBM of some sort. Namely, I would rather maximize youtube premium memberships (which is at "only" 50 millions) over ads (surely they've local-maximized the balance between the two as it is) - but its easier said than done.
I think both are important. Word of mouth is useful and important but no one would use google to search to buy stuff if that was the only way to reach customers.
Also, if your established it probably a good idea not to let new competitors get a foot hold in the market with an easy google win.
It's also pretty effective for local businesses because not a lot of local businesses are tech savvy enough use it effectively.
> people who make them pay upwards of 1 dollar per click
FWIW, the cheapest (quality) clicks I've seen, at least in the B2B space, is closer to $3/click, and it can quickly balloon to upwards of $10/click especially on company brand names where competitors are bidding on another company's brand name.
Knowing this, I cringe every time I'm screensharing with someone and they search "[B2B Company] login" to login to a tool they use every day. Each login = $2-$10
It's not uncommon for companies to spend $100k+/year JUST bidding on their own company name.
It honestly escapes me how these companies can be sustainable. The whole market is sooo inefficient. Companies also pay crazy money to appear in privileged positions in supermarkets shelves, and they will often pay crazy money for simply being in the supermarket at all
I just don't get where all the marketing money is coming from. Bootstrapping is clearly not an option these days
Computers DOUBLED the productivity of the USA since the second world war. All that money went to a few people and groups, and none of it went to average people. For decades, companies have just been sloshing the same giant pile of cash around and around the Ads ecosystem.
That bag of chips did not cost $4 to make, not even a little close.
My working theory is, that advertising is the overhead cost of doing capitalism. There is a certain percentage of resources which have to be spent on advertising to keep the system functioning. Google is good at grabbing a large portion of a huge pile of money.
Not really. It's sufficient to show cool products in "TV" shows (robotic vacuum cleaner in a procedural crime drama might even be a plot device, absorbing murderer's hair to be found by detectives, gasp!).
Coupled with a magazine or a show presenting new product categories for those interested, customers will eventually visit a physical or online shop and check out the goods. And then word of mouth will do the rest.
Aggressive advertising will mostly just help you get ahead of your competitors and perhaps speed up the adoption rate at the cost of increased volatility of the market and to the detriment of people's mental health.
We would be better off regulating aggressive ads away.
> Aggressive advertising will mostly just help you get ahead of your competitors
That's a hell of a load-bearing "just" you managed to insert there. Getting ahead of your competitors in market share can be the difference between having a company succeed or fail.
So if nobody is "getting ahead of competitors", does it mean that "capitalism is not functioning"? (which was the point of the comment to which the reply was)
Product placement is still advertising, likewise advertising plays a role in getting people to go to that online or brick and mortar shop instead of some other one.
I propose a law: nobody can advertise a product without mentioning all the brands which offer same or similar product on the market (and the mention must be neutral or positive).
Or: all advertisers of all brands with a same or similar product must collaborate. Only voluntary input counts as collaboration; if a brand simply doesn't care about presentation of itself in the advertisement, they have trivially collaborated. Easiest way to implement this is giving every owner of all relevant brands a right to veto every entire final advertisement product (this right could also be surrendered, for all or some possible vetoed advertisements, in exchange for something in a contract).
Ignoring flaws of this proposition itself, what could be society's reasons for rejecting it? Does society perhaps want havers of more money to gain further advantage over havers of less money?
>nobody can advertise a product without mentioning all the brands which offer same or similar product on the marke
Maybe 50 years ago that would have worked. Today, not so much. Go to Amazon and look, well, just about anything. What is BEHENO, what is DINGEE, what is Etoolia, what is Romedia, what are the over 300 different 6/7 letter companies that show up when I search up some random product.
Unfortunately your consideration causes its own parasite effect of countless companies forming up to feed of the big advertisers budget.
Since the product is standard, why is it actually bad? If there are too many brands to be included in a single advert, just choose randomly (the lower the price, the higher the probability for a single brand; I don't know the function).
Because, in the US, this will quickly fall foul of free speech laws. Over 'public' airwaves maybe you could go some distance with this, but advertising on private property, as long as it is not fraudulent will present a constitutional challenge to what your saying.
And, you're also crating a regulatory nightmare. Say I put up an add for XXYZXX company, and it includes ZZXYZZ and YYXZYY information (I mean totally random picks), and I just happen to have a stake in those companies too. Now you're going to have to track hundreds of thousands of these entities to ensure no fraud is occurring, and in most cases the fines for this kind of behavior are well under the cost of doing business.
Everything you've said so far just creates bigger messes and solves nothing.
It solves a hypothetical skew towards brands offered by already richer businesses.
About regulation, how hard is it to just audit the random picking procedure?
I now understand that my second variant, with vetoing of final advertisement, is very flawed (one can cheaply obstruct anyone's advertisement by making a company that vetoes any version of it). How about dividing an advertisement into pieces of information solely about each distinct brand, and let every brand owner compose the piece for its brand? Then all pieces are added into final concrete form in a collaboration - I think it would succeed in most cases, and if brand owners can't collaborate, then an independent company will work on it.
Then we need to look how exactly freedom of speech is defined. If it means ability to express views without attaching any additional information, then such freedom is incompatible with my proposal. But if freedom of speech allows attaching additional information as long as base message is preserved, I see no problems. Note that the proposal essentially just forces you to advertise other brands as they wish, along with any advertisement that you do, which (brands) it doesn't mention.
@h4kor one of my crazy ideas is to cap money companies are allowed to spend on Marketing once they reach a certain size. It would encourage a better form of decentralized capitalism and prevent monopolies
This could easily turn out to be counterproductive. It would provide an additional incentive to hide marketing in all kinds of other business activities rather than openly advertise what's on offer.
Marketing is already difficult to tell apart from other company communications, product documentation, etc. What about a company blog showing how to use their products? Is that marketing or product documentation?
The point is that "openly" advertising would be capped. That would reduce the price of doing so, making it more affordable to smaller players and removing the insane profits ad monopolists enjoy today. Plus, "openly" advertising is one of the most effective ways of advertising. Lastly, by diverting marketing budgets to non-traditional routes (charity donations, etc), the economy would benefit as money would be spread more evenly across
I wonder if there’s some sort of automatic stabilizer that could be applied instead.
Tax ad companies, and spend that money on education. The better ad companies are doing, the more we spend on education, the fewer gullible marks we produce, the worse ad companies will do.
sure, but what do you consider to be an ad company? is a newspaper that places sponsored articles an ad company? accounting for "marketing" expenses might be easier to track and at the end of the day, companies use accountants that are liable and so need to report accurately
Exactly, it’s the mechanism for exchanging information in a capitalist economy.
Conversely, in Communist systems they could never get this right. Factories were just told to produce 5 or 10% more than last year, didn’t matter if the product quality was worse or if people didn’t want it.
There was some competition amongst consumer goods producers and TV and other ads in the UUSR. High scarcity of good quality stuff meant they didn’t need to advertise but there was also an oversupply of junk nobody needed. Those companies has to move their inventories somehow since it was much harder for them to go bankrupt.
A little freaky when you think about what that really means. Some of the most advanced AI systems in the world are solely focused on being good at manipulating human behavior. Cool... cool cool cool............
Tangentially, I think this explains the conspiracy theory that ad companies are spying on everyone's phones and serving ads based on what we talk about in real life.
Think about all the stuff ChatGPT and GPT-4 can do with even minimal prompting. Even when they hallucinate, the text is still ostensibly coherent and natural sounding. Now imagine a similarly powerful model, but its input is a ton of metadata about your behavior and its output is ads.
Now consider that adtech has had substantially more funding for substantially longer than research into LLMs, so ad serving models are probably way more powerful and optimized than even GPT-4.
Another thing is: people's individual behavior is not as unique as we'd like to think. As a whole everyone is unique, but in single surprisingly complex aspects of our life we are hardly ever alone.
It's ads that makes the market efficient. Potential customers should know the corresponding producers so that the information assumption of a ideal market stands.
How else would I know that “Elon Musk created a TeslaX platform that allows everyone to get rich”? Or was it Pavel Durov… Seriously, I can’t even report these on YouTube.
There is a mythology to Google's TPU that is not validated by real world numbers. Where we can actually test (I mean -- TPUv4 pods are available right now on their cloud) it is very good, but remains uncompetitive with the h100. I mean, Google disclaims that you shouldn't compare it, doing the classic "the h100 is on a better process node so it's unfair". People will always point at a mythical next generation that is surely way better, despite the fact that Google is currently building big supercomputers with their TPUv4. And in Google's shootout, again comparing with the last generation of nvidia hardware (the A100), Google's biggest advantage was in the connection fabric, which with this DGX GH200 nvidia not only overcame, but bested by a magnitude.
More competition would be fantastic. Better pricing at scale would be fantastic. But there is absolutely no doubt that nvidia is far ahead of Google right now. Tesla made some believably pushing claims about their own efforts with their own hardware, so who knows maybe they're the real challenger.
To add to the other answers, TPUv4 was not released to cloud customers until last year. And I bet availability is not as good as GPUs, even in Google Cloud (obviously TPUs are not available at all in other clouds).
Google has advertised that they have better perf/$ than GPUs, is this wrong or do you just mean absolute cost (so not available in small enough slices)?
edit: actually now i can't find the claim, maybe i misremember what the papers said.
Perf/$ where $ is what it cost _them_ , not $ they're ready to sell to others as a product. Cloud margins in the high two-digit percents are typical, and I'd imagine even higher for very specialized products in high-demand from deep-pocketed customers.
From a personal use case, the number of instructions available in TPUs are still limited and some workaround is needed when designing custom layers. Even if it's available in platforms like Colab or Kaggle, people still lean to GPUs as it is more versatile.
None. Heck, I can’t even search my gmail effectively any more, so if they can’t maintain a core product, I doubt they can build a new one of any quality. alphabet are now just a big, bloated catch-up corporation running on inertia and past glory.
>None. Heck, I can’t even search my gmail effectively any more
Their search products have actually gotten worse with AI. Google Images running just off basic image recognition (as in is this the same image) and the context of where they found it was far superior at identifying what an image is than ML Google Image.
The OG version could identify a frame from a movie and provide higher res versions. The ML version goes “errr looks like a woman on a street, here are random photos of unrelated women on unrelated streets with maybe a similar color scheme” close to useless why would anyone want that. Yandex Image search blows it out of the water simply by being Google Image Search from a decade ago
This is the kind of stuff that I see as being the crux of their downfall. Snippets have also gone to pot over the last year or so.
The overall theme is that product is no longer the focus, but rather navel-gazing - that’s to say, their internal world no longer aligns with the external world, and that is a fundamentally dangerous place for a business.
Yes yes, and the East India Company will reign supreme for all time, Refco is too important to fail, Blockbuster will dominate home entertainment forever, and it’s simply inconceivable that a single trader could bring down Baring Brothers, they’ve been going for centuries!
Businesses fail. google will likely still exist, but alphabet, I don’t see a future for - just a gradual withering followed by a collapse and disintegration into myriad properties in a fire sale. They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more. Inertia will keep it all going for a while, but not forever.
Sure, I may be wrong, but I do put my money where my mouth is, and I am right more often than not.
Your reply is interesting because you strongly believe alphabet will fail but only supported that by arguing that over the very long term so companies fail.
I see a lot of hate for alphabet on HN. It seems very emotional. I think people feel personally betrayed by thier bad behaviours because they were 'supposed to be better'.
The thing is, there are a lot of companies you can hate. Exon, mcdonalds, blackrock, even Microsoft, there are people who are very mad at these companies.
That's not an argument that the company is doomed. If you are really putting your money where your mouth is (what shorting google?) Then I hope you have a better reasoning as to why they will fail not just eventually but this year.
I don’t hate alphabet - neither do I love them. I look at them through the lens of history. You on the other hand seem to be emotionally wounded by my assessment of them.
None of the companies you list are likely to collapse soon, as they remain focussed on their various missions, and have a unity of purpose. Out of all of them, I think Microsoft is the most likely to fail, as they are likely to be blindsided when the user-focussed desktop OS era ends. Their diversification efforts have been a mixed bag, and without windows, they are far, far less significant.
What I do look at is sentiment analysis - what other people feel and think about businesses, as that drives the market.
No, I don’t short, as just buying equities which are beginning significant growth is just as effective and doesn’t drive demise - I held goog for nearly 20 years, and sold off late ‘21, as I think they’ve peaked, and anything from here on is speculative froth.
You’ll note I keep saying “I think”, rather than making statements of fact - because this is purely what I think - I am not a Sybil.
You seem to have missed this:
>> They are brittle, overburdened by unity of disparity, culturally adrift, and they aren’t taking risks any more.
> Out of all of them, I think Microsoft is the most likely to fail, as they are likely to be blindsided when the user-focussed desktop OS
That might have been a reasonable assessment back in Balmer’s era. But what you saying has already happened years ago…
They have mostly reinvented themselves since then. Enterprise/office isn’t going anywhere. Xbox if fine too. And there is a lot of growth in their cloud/etc. business.
IMHO out of Google, Amazon & Facebook, Microsoft seems to be the least dysfunctional and and general best positioned one to be successful in the future.
Xbox doesn't seem fine. I think it's propped up by game pass having cross platform title access with windows but it's still under the Xbox balance sheet, but growth and number of exclusives doesn't paint a healthy picture.
Yeah by Xbox I mean the console + game pass + PC/Xbox gaming division. The console itself at this point is not much more than a cheap(ish) locked down gaming PC.
For context, I have never worked for or with google, and don't use their products much other than search. So I don't have much emotional connection to the company. My comments were more motivated by a kind of concern.
My perception is that Google split into a number of focussed business units when they became Alphabet, with the Google component being execution focussed and the more speculative stuff spun out into other group companies like deepmind, waymo, etc. That's why the Google unit stopped doing nice incubator projects that we were all excited about.
From what I've seen, this cash cow execution business unit has been fairly effective - in particular they've done a good job of entering the cloud market space producing a differentiated product that is penetrating their target customers. They have not been able to compete with Microsofts excelent and deeply embedded IT sales capability, so they've done well to go after people with big problems that other vendors more civillian offerings are not so great for. They are currently the first choice platform for AI training for instance.
I'd contrast this to Facebook who seem to be trying to become a deep tech VR hardware vendor in the same business unit as their cash cow entertainemet and advertising business which has confused investors and probably distracted their focus.
We can see that Google has innovated. For instance, a lot of Tela's stock price is based on the idea that they are going to run autonomous taxis, and instead of owning cars we will just hail a Tesla when we need one. Telsa does not run autonomous taxis, but you can ride a Google Waymo taxi today in Pheonix, and they are running autonmous trucks which is a big industry Tesla aren't even attempting yet. They are doing a lot in medicine and medical devices. This seems a lot more diversified and innovative than other companies - it's just not as visible to the HN community as an RSS reader or some other internet thing we care about.
We can also say that... on the AI thing, I think it's very early days. Microsoft have a shakey looking deal with the first mover, but Alphabet and Facebook have the advantage of actually using AI extensively in their real buisnesses and may be able to deliver product market fit better. Time will tell.
On the stocks front, I agree with your overall thesis - I think it's harder for these conglomerates to grow than a new company just because they are already giants in their niche and even adding a new niche generates less growth in percentage terms than for a smaller company starting from a lower number. I just wouldn't actually bet against google as much as I would some of the others.
you originally said 10 years. so hopelessly delusional lol. since you're so confident let's bet $10,000. By your claim let's bet by 2034 (I'll give you some extra time). Alphabet Corporation and all subsidiaries will no longer exist. If they do I get your $10K. If it does not you get my $10K.
We can both give the money to a mutually trusted third party now.
If you believe this it implies you have gone full malthusian, because the innovation engine that made Google possible could also disappear Google, but without that engine we are all screwed.
gmail is a freebie! the core product is how they index your messages to create an anonymous profile that they will then offer on reverse bid to advertisers when you do a search or visits an AdWords site.
If you don’t match the terms in your email exactly right with your query, gmail starts returning email that matches one of the terms which is rarely what you want.
What would you expect it to do? How is it supposed to know what you want if you don't provide it with the exact search terms? Shouldn't it do partial matches if there are no full matches ?
The GH200 uses a combination of HBM3 for the GPU and LPDDR5 for the CPU but it's a unified memory system so the GPU can access all the RAM. Gaming GPUs use GDDR which is a third flavor.
As always in economics it is about volumes and margins.
If the competitors (mainly AMD, Intel and to some extent ARM) will keep seeing growing volumes and insane margins they will be attracted to bring to invest and take part of that market.
Till now gaming GPU market did not bring to AMD the necessary margins to really push them to bring a better competition to Nvidia. Even 10/15 years ago when ATI was way ahead of Nvidia technologically for 2/3 years (the HD 4000 and HD 5000 generations vs the Nvidia flops of the 9000, 200 and 400 series) Nvidia was posting billions of profits and ATI posted a whole...19 millions of profits across 3 years.
But today's GPU market thanks to it's non-gaming sales is much bigger to ignore (which is why Intel entered it as well) and those players will likely react.
You don't need to have the best premier product, you need to have your products good and priced well enough that they will be chosen over the competitor's.
In hindsight the 40GB of SRAM feels kind of quaint, but nevertheless their very fat nodes let them get away with more than Nvidia could with A100s, as you can see in the slides.
CS2 is a little old now. I bet an update is just around the corner.
I have to wonder why these engineers are not paid millions of dollars per year. As a lowly backend dev this seems so much more impressive than my new API that retrieves something from a database...
Isn't this a lot of things which AMD has already sold as the ORNL-Frontier 2 years ago? The main difference seems to be that external bandwidth here indeed is crazy via NVLink (though it is only 450GB/s per way so the same as 64 PCIe-Gen5...) and they have two networks for communication (although I suppose the HPE Slingshot is as good as the Infiniband in here)...
Sounds like you havent seen Wafer-scale integration computing, Tesla has one and comercial companies like cerebras will sell you a cabnet without the miles of fiber networking.
I'm curious how you can keep that fed with data fast enough. What kind of interfaces to your network do you need to keep it busy and not just waiting on data.
Ultimately how fast their transistors can switch and at what power is determined by TSMC, which everyone else can use too. Same for density of interconnect.
Is there really a market for a response though? Now, I'll be honest that I know very little about this market. What I do know from doing a decade of presales before covid hit, is that people who buy GPUs go for aggregate max on a big node farm. Now, most of my clients who bought GPU-heavy scale-out nodes were in the financial industry, so maybe deep learning stuff is different. Their workloads were massively parallel, and could scale out instead of needing something singularly fast.
So I guess my question is - what use case is there for a huge truck that goes 200mph and take 4 trips, when you could just buy 16 regular trucks, and move your apartment in the same amount of time at half the cost.
The exciting thing about CXL is we can start to find out if peripheral or hopefully close-networked computing fabrics can be useful & interesting, beyond the small circumstances Nvidia will offer. Having an ecosystem that everyone can participate in will let us explore. Money can't buy that. Talent can't buy that. You need to socialize to really find out the possible values.
'The street finds it's own uses for things' is the well known Gibson adage, I and typically it's a comment aimed low. But our entire era of amazing computing began with the Gang of Nine enabling lowness in a degree such that it quickly became the highest tech, the best. Sure you can still buy a mainframe & they have amazing feats but it's not where the value is, but and the value is where it is because possibility was unchained, I unleashed from corporate dominion, and spread wide. I think we can find amazing new futures with CXL & mad bandwidth connectivity.
The reason that analogy falls short is because it's easier to drive the huge truck at 200mph than it is to find 16 truck drivers. It's really neat when you figure out how to map/reduce your algorithm so you can parallelize it, but it would be even easier if you didn't even have to in the first place. And that's assuming that it is even parallelizable in the first place. Not all algorithms can be optimized like that and needs a bigger system to run on.
There are workloads that are data parallel, and scale like the GPU-heavy scale-out nodes that you describe.
The other approach, which you do when models themselves are massive, is model parallelism. You split it into multiple parts that run on different nodes.
In both cases, you need to distribute weight updates through the network although the traffic patterns can be different.
To maximize the performance in both scenarios, systems designers optimize for all-reduce and bisection bandwidth.
There are also other tricks, for example the TPUv4 ICI network is optically switched, and it is configured when a workload starts to maximize bandwidth for the requested topology ("twisting the torus" in the published paper).
Using something like Stable diffusion and generating all the frames at once (for video) as a single image. For that kind of usage one needs to have ram for the whole image. This setup could generate videos like that in the same time as I generate an image on my home computer.
How the rest of the industry can respond is such a mystery. And will it be lone competitors, or will a new PC era be able to start, with an ecosystem of capabilities?