Nvidia’s New Rubin Servers Offer Early Relief to Cloud Providers
Finally, there’s good news for Nvidia customers who have been contending with more and more difficulties getting data centers up and running: Nvidia’s next-generation chips probably won’t be one of them.
That’s according to several executives at cloud providers that purchase Nvidia AI servers. They told me that based on their initial tests, they expect the company’s upcoming Vera Rubin server racks to be much easier to install than Grace Blackwell racks were in 2025. Back then, Nvidia’s biggest customers struggled mightily with an array of new technologies they hadn’t dealt with before. Their struggles were kept quiet, at least until my colleagues reported on it.
For cloud providers spending billions on hardware, every week spent troubleshooting new servers would delay bringing on lucrative compute capacity for AI customers. Setup delays could also stall orders and squeeze margins for data center builders.
It’s still early going with Vera Rubin racks, which are just beginning to trickle in to customers. The real test will come when customers begin to receive hundreds or thousands of Vera Rubin racks next year.
By the second half of next year, Nvidia aims to produce a more powerful version of the rack, which could present new problems for customers because it will require a lot more energy to run as well as more switches and cables to connect servers to one another.
For now, the racks customers are testing—each of which contain 72 graphics processing units and 36 central processing units—are essentially a third-generation product for Nvidia, not a brand new one. That’s because Grace Blackwell racks had a nearly identical configuration. (The more powerful versions of Vera Rubin racks, which link 144 or 576 GPUs together, aren’t being produced yet.)
“Honestly, we don’t think it will be too difficult” to install Vera Rubin racks at scale, one of the cloud executives said. “Mechanically and thermally, it is now a third generation product.”
Nvidia says Vera Rubin systems are fairly compatible with the software customers already use to run Grace Blackwell servers. Two cloud executives said it would still take a few months to adapt their companies’ software to the Vera Rubin server racks to run them in large clusters.
‘Black Magic’
Vera Rubin racks cost at least twice as much as the initial Grace Blackwell racks did, but Nvidia says the new hardware is 10 times more efficient, based on a metric known as AI tokens per second per watt.
To be sure, Vera Rubins won’t be a cinch to install. Many of its components are brand new, one of the cloud executives said. And the Rubin racks have 1.3 million components, compared to 1.2 million for the initial Blackwell racks.
An Nvidia Vera Rubin NVL72 compute tray, which includes four Rubin GPUs and two Vera CPUs. Source: Phoebe Liu
In March, Nvidia CEO Jensen Huang alluded to the complexity of some of those components when he showed off the new networking switches that connect the new Rubin GPUs to each other. “This is insanely hard to do well. This is insanely hard to do, period,” he said at his company’s annual developer conference.
Huang didn’t say whether the new chip-to-chip networking would be difficult for customers to handle, but such technology slowed the Blackwell ramp-up last year. An Oracle employee who works on large computing clusters, including for upcoming Vera Rubin servers, said that figuring out the location of a networking failure in such clusters was excruciatingly hard, requiring “black magic.”
Rubin racks also consume 75% more power per rack and run hotter than the Blackwell predecessors. Counterintuitively, the new cooling system in the rack is supposed to consume less power than cooling technology for Blackwell racks but it’s harder to set up. One analyst who tracks Nvidia’s stock closely said thermal engineering issues had set back the production of the initial batch of Vera Rubin servers by about two months. An Nvidia spokesperson said that “nothing has changed from what we already announced.”
On top of that, getting Vera Rubin online doesn’t just depend on the Nvidia hardware. Many customers are building new data centers with Nvidia-recommended floor plans, cabling setups and management software to house the Vera Rubins efficiently.
Data center developers “have to build brand new infrastructure from the ground up” for Vera Rubins because retrofitting existing data centers would be too costly, said Kevin Cochrane, chief marketing officer at Vultr, a GPU cloud provider.
Ultra Version
Another test will involve the more powerful Vera Rubin rack, Ultra, which is slated to appear next year. Unlike the first Vera Rubin racks, the Ultra ones will have several new features that could take some time to figure out. Trays containing the chips will slot into server racks vertically like a bookshelf rather than horizontally like a baker’s rack, which means gravity could put a different type of stress on the new system.
The racks will also link as many as 576 GPUs together and need nearly triple the amount of power per rack, compared to the less powerful version, which could all present new challenges.
Andrew Bell, Nvidia’s senior vice president of hardware, said earlier this month that Blackwell “broke everything” because Nvidia completely redesigned the GPU system, making life difficult for customers.
By comparison, Vera Rubin is a less jarring change and “way easier to manufacture,” he said. There’s a 95% chance of getting a tray of Rubin chips assembled correctly on the first attempt, compared to 20% for the first Blackwell systems, he said.
Not everyone is a believer. One cloud executive was skeptical of Bell’s claims, saying they expect the advanced Vera Rubin systems to “break everything” again and be as difficult to install as Blackwell racks were.