Oracle Exec: AMD Gaining Favor in Nvidia-Dominated Market for AI Chips
The Takeaway
Karan Batta, a top Oracle cloud executive, says competition with Nvidia is necessary to reduce hardware costs—and control expenses for businesses that want to make their own AI apps by running their own large language models.
Despite Nvidia’s booming growth in selling data center chips for artificial intelligence, some customers developing AI are turning to chips from smaller rival Advanced Micro Devices, said Karan Batta, a senior executive in Oracle’s cloud business. Oracle rents out chips from both companies.
“What we’re finding now is customers are not necessarily tied to any one particular vendor for inferencing,” Batta said. Inference computing refers to AI that has already been developed and is powering applications.
“There’s a lot of effort being put into AMD, and [the company is] doing a good job” at improving its chips’ performance, he said.
Batta declined to provide specifics or names of customers, but his comments show how Oracle wants to reduce its dependence on AI chip leader Nvidia, similar to Microsoft and other cloud providers.
While AMD’s latest flagship server chips, launched in 2023, may not be as powerful as the latest Nvidia flagship—the H100, launched in 2022—executives at Oracle and Microsoft feel they have little choice but to support AMD to promote competition.
In the meantime, Oracle and other cloud providers are booming as they rent out more Nvidia servers to customers such as OpenAI and Elon Musk’s xAI. Oracle is a distant fourth place behind Amazon Web Services, Microsoft Azure and Google Cloud in server rentals in the U.S. It generated $2.4 billion from that business in the August quarter, up 45% from a year earlier.
Oracle entered cloud computing relatively late but got somewhat lucky last year, as its business benefited from a frenzy among AI startups to secure cloud servers. It also has an advantage with some AI developer customers because it’s not a competitive threat to them; Oracle, unlike its bigger rivals, isn’t developing its own large language models similar to OpenAI’s GPT-4.
The boom in AI server rentals reflects the development or training of new AI models. But many businesses have yet to adopt LLMs aside from using applications such as ChatGPT, GitHub Copilot and a handful of others in areas like customer service. Those businesses’ leaders have said the technology is not useful enough or costs too much.
As Nvidia runs away with the server chip market, competition is needed to drive down hardware costs, said Batta. Doing so will keep AI inference costs under control for businesses that want to use LLMs to make their own AI apps or want to buy such apps from software vendors, he said. The cost to rent Nvidia chips for inference has been dropping, but its next flagship AI chips might change that.
Other Nvidia customers have expressed similar concerns privately, reflecting fear that Nvidia products will continue to command a large premium if AMD or others don’t eat into its share of the market. So far, that hasn’t really happened. AMD said its data center revenue rose 115% to $2.8 billion in the June quarter compared to the same period a year ago, while Nvidia data center revenue rose 154% to $26.3 billion in the July quarter.
“Nvidia innovates constantly to deliver enhanced value to our customers, enabling AI for every cloud and enterprise,” an Nvidia spokesperson said in a statement.
Oracle’s use of AMD hardware could also help it guard against potential Nvidia supply shortages, Batta said. Such shortages occurred throughout much of last year.
As Oracle looks to capture more of the market for powering AI model training, co-founder Larry Ellison said Monday that it is designing a data center with a capacity of more than one gigawatt, enough to power a city like San Francisco. Batta declined to provide details on that data center or when it might be operational.
To support its expansion, Oracle is investing in its ability to string together clusters of AI chips so they can act like a single computer—the ideal setup for training new LLMs, he said.
Those improvements include changes to how Oracle cools the heat-generating chips and builds tunnels for cables to link chips across different buildings so they don’t all need to be in the same place. Oracle plans to at least double the size of the interconnected groups of chips, known as clusters, it offers customers—from 64,000 graphics processing units to more than 128,000, he said without specifying the timeline.
Oracle can still benefit from demand for AI training even if startups are using other clouds to develop their core models, Batta said. For example, companies also need hardware for other purposes like fine-tuning, in which developers hone a model for a specific task. Some AI companies are buying access to older chips for those workloads, Batta said.
That might be a reference to a deal Oracle struck with OpenAI and Microsoft to use a facility in Texas next year, according to the AI Data Center Database, but Batta declined to comment on specific companies.
Oracle recently faced a setback when Musk decided to build his own GPU cluster in Memphis rather than rent one from Oracle because it couldn’t move fast enough to set up the data center. Batta said it was too early to say whether xAI can get its Colossus cluster of 100,000 H100s completed faster than Oracle would have.
“I don’t have a baseline to compare with. Am I way faster [than xAI]? Am I way slower?…I do think if you compare us with other cloud providers, we feel like we’re one of the fastest, if not the fastest, from an AI compute perspective.”