What it Takes to Make Open-Source AI Cheaper Than OpenAI; Microsoft Goes Multimodal
When Meta Platforms’ released its open-source Llama models earlier this year, the tech giant was praised for giving developers a free alternative to the expensive, bleeding-edge large-language models offered by proprietary LLM developers such as OpenAI.
Turns out it’s not so simple.
In many cases, using open-source LLMs can actually be more costly than proprietary AI software from OpenAI and its ilk, as I discovered while researching this piece I published this morning.
That’s because OpenAI and Meta are providing essentially different services. When companies use OpenAI’s model, it takes care of all the pesky behind-the-scenes work of building and running a model. All users have to do is tap into their nicely-packaged application programming interface. In contrast, anyone relying on an open-source model like Llama starts with a bare-bones model. They then have to figure out the complexities of actually getting that model up and running, which can end up being expensive.
Luckily, there are a few things startups can do to use open-source AI models without breaking the bank.
First, not every task requires a state-of-the-art model. As I discussed in my piece, smaller models can handle simpler tasks like summarizing and translating text, while reasoning-heavy applications like code generation are best-suited for the latest LLMs like GPT-4. (And there’s even more wiggle room within that. For instance, research has shown that open-source code generation models like Meta’s Code Llama can be greatly improved by training them on model-generated coding examples—crazy, I know!)
Some of these smaller models, like the 7 billion parameter version of Llama 2, can run on older-generation AI chips, such as Nvidia’s V100, that are cheaper and more widely available than the chipmaker’s in-demand H100s, said Vikram Sreekanti, cofounder and CEO of model deployment startup RunLLM. (In case you’ve forgotten, “parameters'' refers to the “settings” of a model.) Even better, these smaller models are faster to run, making them optimal for the generative AI features that customers use the most.
These models can also be optimized further through a couple different approaches. One is called “quantization,” a technique that simplifies the parameters of a model. In most cases, it can significantly reduce compute needs with only a slight hit to model quality.
Another is “batching,” in which model queries are bundled together and processed simultaneously rather than one at a time, which uses the AI hardware more efficiently. Unfortunately, most companies can only really take advantage of that hack once they reach a certain scale, as it requires a consistently high volume of model requests (something that OpenAI and its peers benefit from).
An additional way developers can improve their model’s performance without having to spend too much is by training them on their own data, otherwise known as “finetuning.”
But Stephanie, you might say, OpenAI offers finetuning for GPT-3.5 Turbo too! Sure, you’re not wrong there. However, OpenAI’s finetuning uses a technique called low-rank adaptation of large language models, or LoRA, according to a person with direct knowledge of the product. LoRA is cheaper but slightly less effective at improving a model’s performance because it tweaks just a subset of a model’s parameters versus full-parameter finetuning. So if you’re a startup that wants more control over model quality, perhaps opt for full-parameter finetuning on an open-source LLM instead.
The Takeaway: If you’re a startup that’s just getting up and running, it may make sense to start out by experimenting with OpenAI’s proprietary models, especially since they handle the hassle of optimization and batching for you. However, once you’re able to grow your customer base, open-source models could provide a cheaper and more customizable alternative. (Plus, there are plenty of companies out there, like OctoML and Mosaic, that can help you with model optimization.) And if you’re lucky, the combination of optimization and batching could land you an open-source model that runs at less than 1% the cost of a finetuned version of GPT-3.5 Turbo, according to one startup I spoke with.
Here’s what else is going on…
Microsoft Dips Its Toes In Multimodal
A few weeks ago, we told you that Microsoft’s research division had a new priority: building AI models that are smaller and cheaper than OpenAI’s GPT-4 and that could still mimic the larger model’s quality. As we reported, one of Microsoft’s priorities was to make those smaller models “multimodal,” meaning they could understand images in addition to text.
On Wednesday, Microsoft researchers confirmed that they’ve upgraded one of their in-house AI models, dubbed Phi, to be multimodal. That breakthrough has big implications for developers who want to build AI into their apps but don’t want to pay a premium for state-of-the-art models like GPT-4. Microsoft Research has said that it eventually plans to release Phi as an open-source model, meaning anyone would be able to use it. And Phi is just a fraction of the size of OpenAI’s models, with 1.5 billion parameters compared to GPT-4’s several trillion parameters.
The proliferation of smaller models like Phi could chip away at OpenAI’s dominance. Microsoft product teams have already been switching out GPT-4 for open-source models to carry out more rudimentary tasks in products like Bing, The Information first reported. And companies who initially paid top dollar for OpenAI’s models have recently been eyeing a rising tide of cheaper competitors, including open-source models.—Aaron Holmes