TheChek News Flow

The stumble meant OpenAI had lost precious time and would need to shift its resources toward developing a different model. The failure also disappointed some executives at Microsoft, which paid for the right to use the startup’s technology in its products, according to a Microsoft employee with knowledge of the matter.

The Arrakis setback could pierce OpenAI’s aura of invincibility after it humbled AI pioneer Google and built one of the fastest-growing software businesses in history. It shows how the frontier of AI is riddled with pitfalls that can be hard to predict.

Although the Arrakis problems didn’t slow OpenAI’s business this year, the startup could feel the effects in the coming year as the race to launch new LLMs intensifies. Google, for instance, is nearing the launch of Gemini, a set of AI models it hopes will beat GPT-4 in terms of coding and other capabilities, and the accuracy of responses. OpenAI, for its part, continues to release improvements to its industry-leading model, including the ability to decipher images, and plans to announce a slew of new features in November. LLMs underpin products such as ChatGPT, and some at OpenAI also view them as having the potential to be a kind of operating system, including for personal devices, because of their ability to write code, make sense of images and retrieve files.

Based on the strength of GPT-4, OpenAI’s business has ballooned. The company is generating $1.3 billion in annualized revenue, up from $28 million in revenue for all of last year, on the back of GPT-4, which powers the paid version of ChatGPT. OpenAI is worth around $30 billion on paper after an employee share sale in the second quarter of this year, but the company is trying to boost that valuation considerably as it works on another tender offer for employees.

OpenAI started working on Arrakis last fall in the hopes of developing a model that would be on par with GPT-4 but could run more efficiently, in part by leveraging a machine-learning concept known as sparsity, the people said. Other AI developers such as Google have also publicly discussed their use of sparsity, which OpenAI successfully incorporated in earlier software. Arrakis would have allowed OpenAI to roll out its technology more widely, as the company had access to a limited number of specialized server chips to power its software, they said.

Around this spring, OpenAI’s researchers began training the model, which involves using advanced computing hardware to help the model process massive amounts of data so that it can learn patterns. The company expected it to be significantly cheaper than the process of training GPT-4, the people said. Early on, however, employees realized that the model was not performing well enough to reap the expected benefits. After staff spent roughly a month trying to fix the issues, OpenAI’s senior leadership decided to pull the plug on training it, the people said.

Despite the setback, OpenAI could still incorporate its work on Arrakis into other models. That includes Gobi, an upcoming model that can generate or analyze text as well as visuals, also known as multimodal.

Arrakis underperformed OpenAI’s expectations after the company attempted to increase the model’s sparsity, meaning it would use only part of the model to generate responses and thus make it cheaper to run, two people said. The reasons why the model worked in early tests but didn’t perform well afterwards couldn’t be learned.

Arrakis—named after a desert planet in the “Dune” series—is a nod to the use of sparsity in the model’s design, one person said. Other OpenAI model codenames, including Gobi and Sahara, which was publicly called GPT-3.5 Turbo, use a similar desert theme to reflect the company’s effort to make them sparse.

Since the failure of Arrakis, OpenAI researchers pivoted to developing a version of GPT-4 designed specifically to generate responses more quickly, the two people with knowledge said. OpenAI has discussed referring to that updated model as GPT-4 Turbo, which was also the public name OpenAI considered for Arrakis before it failed, one of the people said.

For OpenAI, making its models cheaper and more efficient is a top priority as concerns grow about the technology’s costs and as open-source alternatives proliferate. An OpenAI spokesperson didn’t have a comment.

Microsoft, which uses OpenAI’s GPT models to power AI features in Office 365 apps and other services, had expected Arrakis to boost the performance and lower the cost of those features, said the employee with knowledge of the situation. Bing, the Microsoft search engine that relies on GPT-4 and other models to power a ChatGPT-like chatbot, was hoping to implement the Arrakis model in early 2023, before the model was completely derailed, this person said.

Since then, Microsoft has worked on other LLMs that could be cheaper to run compared to OpenAI’s, The Information has reported.

Many industry practitioners expect sparse models to bring down AI development costs. At a conference in August, Jeff Dean, Google’s chief scientist and a key contributor to its Gemini AI models, said the industry will move toward sparse models in the coming years. Unlike “denser” models like Llama 2 that power some generative AI apps, sparse models only call upon certain computations within the model, known as parameters, that are needed to complete a task, making the process more cost-efficient.

“Sparse computation is going to be an important trend in the future,” Dean said.

Some AI practitioners say one way to increase sparsity is through a technique known as “mixture of expert models,” in which specific parts of a large model are trained to handle certain tasks—in other words, those sub-models become an expert in those tasks—so that the full model doesn’t need to be triggered. OpenAI incorporated this technique into GPT-4, and Arrakis would have also done so, the people said.

“In general, the larger the number of expert [models] is, the sparser and the more efficient the model is,” said Ion Stoica, a computer science professor at University of California, Berkeley, in an email. The result, however, can lead to less accurate results, he said.