OpenAI Discovers New Way to Cut Inference Costs in Half
We closely track efforts by Anthropic, Google and OpenAI to get access to more server chips to run their models. But we don’t talk enough about the work these companies are doing to get more juice from the servers they already have.
In one previously unreported example, OpenAI engineers earlier this month told some colleagues they had figured out a way to more than halve the cost of inference, or running existing models, thanks to some newly-discovered optimizations, according to a person with knowledge of those discussions.
When the engineers applied the new techniques to power ChatGPT for visitors who didn’t have a free or paid account, it reduced the number of Nvidia graphics processing units needed at one point to just a couple hundred—a shockingly small number. (That said, OpenAI likely doesn’t get much ChatGPT usage from such users, as the company limits how much they can use the chatbot that way.)
It isn’t clear what OpenAI did to get its latest efficiency gains, which might include techniques such as quantization; key value-caching, or helping the model remember information from prior calculations it made so it doesn’t need to repeat the work; sending queries to be answered in batches rather than one by one; and routing some queries to models or parts of models that require less power to answer them.
And the ones OpenAI figured out this month may have less impact later in the year when the company develops bigger models in terms of parameters, the settings that determine its answers, as bigger models tend to cost more to run.
However, these inference optimizations, which rival Anthropic refers to as “compute multipliers,” are a big focus for all the labs. Anthropic CEO Dario Amodei has been publicly talking about the concept since at least mid-2023, when he said on a podcast that the company limits “the number of people who are aware of a given compute multiplier” because it could give other AI labs a leg up if they were to be able to replicate them. (Compute multipliers can also refer to efficiency optimizations in the model-training phase.)
These optimizations are especially important because large AI developers are having a hard time getting access to enough servers. Even when they are able to sign contracts to build or lease new data centers, it can often take months or years to get them up and running. (OpenAI is also trying to further reduce its inference costs by launching its own chip with Broadcom that’s specialized for running models and aims to be cheaper than using Nvidia chips.)