Why AMD’s Upcoming Chips Won’t Be the Savior AI Startups Are Hoping For
Two weeks ago, there was a flurry of excitement around a tweet from Sharon Zhou, cofounder and CEO of Lamini, a startup that helps AI developers create customized large language models. Zhou revealed in the tweet that, for the past year, she’s been using more than 100 chips from chipmaker Advanced Micro Devices to power her AI startup’s products.
In the midst of the ongoing compute crunch, AI founders and investors rallied around the news, seeing it as an indicator that AMD could cash in on the intense demand for AI chips, possibly even taking market share from the main supplier of those chips, Nvidia. AMD stock has jumped 13% since then.
Indeed, the AMD chip in question, the Instinct MI300A, has often been touted by founders and analysts as a strong alternative to Nvidia’s industry-leading H100. It’s due for release later this year. But before AI startups get too excited, they should pay attention to all the details of Zhou’s experience with the AMD chips. The short version: it’s not as easy as it looks to use those chips.
Zhou told me that while her startup officially launched a year ago, her cofounder, Greg Diamos, had been working for several years on optimizing AMD’s chips for the software Lamini has developed. So it would be nearly impossible for a startup that has built AI apps on top of Nvidia chips to switch to AMD chips now because it would have to “throw out all code and start from scratch,” she said.
Nvidia has a “two-decade head start” when it comes to Cuda, Nvidia’s “hardware acceleration” software that helps developers build apps utilizing its chips, she added. And that software only works with Nvidia’s chips and developers are already familiar with it. It likely helped Diamos’ efforts that he was an early developer of Cuda.
That’s not to say that AMD’s chips don’t have any advantages.
The AMD MI300A chip combines both a graphics processing unit, which performs multiple computations simultaneously, and a central processing unit, which executes more general instructions and manages the system’s broader operations. That’s in contrast to Nvidia’s H100 chips, which are GPU-only. (Nvidia’s newer Grace Hopper Superchip, however, similarly combines a GPU and CPU.)
That chip combination provides a few benefits. In model training, the CPU is responsible for preparing and loading data onto the GPU, where the actual computations occur. Placing the CPU and GPU closer together speeds up this process.
Additionally, AMD’s combined CPU and GPU MI300A chip has more memory—128 gigabytes versus 80 gigabytes in Nvidia’s H100 chip. Increased memory capacity means developers can load a bigger, more complex AI model on a single chip versus splitting it across multiple chips. Splitting up the model makes training and running it slower and more power-intensive.
But historically, AMD has been known for serving national labs with AI chips, such as the Oak Ridge National Laboratory, said Bing Xu, cofounder and CEO of HippoML, a startup that makes AI models run faster and more cheaply. Because of this, its chips are optimized for FP64 large matrix multiplication, a highly complex and precise computation used in training models, Xu said. But most startups running AI models require lower precision and higher speed. AMD chips aren’t specialized for this use case and therefore tend to be slower.
The dominance of Nvidia’s Cuda software has also made it difficult for AMD’s equivalent, an open-source software called RocM, to catch on. Nvidia additionally has a huge head start on other software components such as drivers, which connects operating systems to the hardware, and therefore has had more time to iron out lots of bugs in the process.
All of this creates a Catch-22: AMD’s chips and software can only improve if more developers use them, but those developers won’t want to use them if they’re buggy. The chip bottleneck may be enough to push some AI startups towards AMD, but especially given the crazy amounts of capital being thrown around, many may prefer to pay up for what they’re used to: Nvidia chips.
The good news from AMD’s point of view is that a number of startups are working to make it easier to use non-Nvidia chips. Lamini, for instance, is aiming to simplify many of the difficulties of building AI models on top of AMD GPUs. Zhou hopes that she’ll be able to popularize the usage of AMD chips so cloud providers will begin offering them. Meanwhile, Modular is building software to allow developers to train and run models on different types of hardware. (Plus, it looks like AMD itself is interested in snapping up some of these startups: On Tuesday, the chipmaker announced its acquisition of Nod.ai, a startup building software that helps companies deploy AI models tuned for AMD’s chips more easily.)
The combination of these up-and-coming startups and the chip shortage make this a unique opportunity for AMD to overturn its biggest rival—the only question is, will they succeed?