OpenAI Develops AI Voice Assistant As It Chases Google, Apple
In the race to develop artificial intelligence that communicates the way humans do, OpenAI is preparing to demonstrate technology that talks to people—using sound as well as text—and recognizes objects and images. The ChatGPT developer has shown some of these capabilities, which include better logical reasoning than its current products, to some customers, according to two people who have seen the new AI.
The technology is another step in OpenAI CEO Sam Altman’s quest to ultimately develop a highly responsive AI akin to the virtual assistant in the Spike Jonze film “Her,” and to enable existing voice assistants like Apple’s Siri to be more useful. The company could preview the upgraded AI publicly at an event as soon as Monday, which would help it get ahead of a slew of AI announcements from its rival Google later in the week, one of the people said.
The Takeaway
• New OpenAI software has built-in audio and visual understanding
• It could boost the performance of automated customer service agents
• OpenAI could complete GPT-5 and release it publicly by the end of the year
OpenAI sees assistants with visual and audio capabilities as potentially as transformative as the smartphone. The assistant could theoretically do a range of things not possible today, such as acting as a tutor for a student working on a paper or on math problems, or giving people information about their surroundings when they ask for it, like translating signs or explaining how to fix car troubles.
The new tech is too big to run on personal devices today, but customers in the near term could use the cloud-based version to improve features OpenAI’s software already powers, such as automated customer service agents. The audio features of the new software could help such agents better understand the intonation of callers’ voices or whether they’re being sarcastic in making a request, said one of the people with knowledge of it.
OpenAI already has software that can transcribe audio and convert text to speech, but those features are available through separate conversational AI models, whereas the new model brings those features together. That gives the new multimodal model a better understanding of image and audio, as well as making it faster to use than the less-capable models.
Microsoft, which can use OpenAI’s technology at will because it is the company’s top financial backer, could use OpenAI’s new AI to improve its own voice assistant or try to make it compact enough to run on small devices, including wearables with front-facing cameras that can capture the customer’s surroundings.
It isn’t clear when OpenAI will make the new features available to its paying customers, but it eventually plans to make them part of the free version of its chatbot, ChatGPT, said one of the people who has used it. OpenAI aims to make the new AI model powering these features cheaper to run than the most advanced model it sells today, GPT-4 Turbo, this person said. The new model also outperforms GPT-4 Turbo in answering some types of questions, this person said. However, the new model can still make mistakes, known as hallucinations. (Spokespeople for OpenAI did not respond to requests for comment.)
GPT-5 Release
Google executives, meanwhile, have long dreamed of using AI to develop powerful assistants. In December, Google showed a video of a conversational AI it had developed, Gemini, that responded to a person’s voice commands in real time and recognized images the person was looking at. However, the company separately explained that these capabilities required researchers to prompt the models with images and text instructions, rather than the simple dialogue the video demonstrated. In the meantime, Gemini has added features that can analyze audio in addition to imagery and text, but it doesn't understand many traditional voice commands or talk to users the way traditional voice assistants like Siri and Google Assistant do.
Still image from Google's December demonstration of multimodal features of Gemini AI, via YouTube
OpenAI also is trying to stay ahead of Meta Platforms, which in April released an open-source AI, Llama 3, that surpassed the performance of most conversational AI models available today and received rave reviews from AI app developers.
The upcoming OpenAI model with audio and visual capabilities is one of a number of products under development. The company has been aiming to launch a web search engine, which aims to compete with Google’s. (The Information first reported on it in February.) OpenAI also is developing a type of automation software known as a computer-using agent that could speed up software development and other computer-based tasks, and the company has previewed an AI video generator, Sora, that isn’t available publicly yet but has made waves in Hollywood.
More importantly, OpenAI has been developing GPT-5, which it hopes will represent a significant improvement over GPT-4—a model it released more than a year ago. It could complete GPT-5 and release it publicly by the end of the year, said a person who has discussed it with OpenAI leaders.
The blitz of product and AI model development at OpenAI means some projects previously announced aren’t getting as much attention. For instance, though the startup promised developers that by the first quarter of this year they would be able to make money from building custom chatbots for its store, OpenAI has yet to launch a way for them to do so.
On the other hand, improving visual and audio capabilities could aid OpenAI in getting its conversational AI running on millions or billions of Apple devices. The iPhone maker has held discussions with OpenAI in recent months about how the next iPhone operating system could integrate OpenAI’s models, Bloomberg reported. However, the ChatGPT maker has tough competition: Apple is holding similar talks with Google, Bloomberg reported.
Altman is also working with iPhone developer Jony Ive on a separate AI consumer device, which could raise up to $1 billion in funding from investors including Emerson Collective and Thrive Capital, The Information first reported. In doing so, Altman would be joining the ranks of the big tech companies and startups all racing to release AI-powered devices and wearables that could capture the imaginations—and wallets—of consumers.
However, the large size of the most advanced AI models means they will need to run in the cloud for now and require an internet connection to work. It could take months or even years for complex conversational AI with visual and audio capabilities to become small enough to run on devices.
Tiered Pricing
OpenAI, which could generate billions of dollars in revenue this year, is also planning to release a new pricing model that would offer customers up to a 50% discount if they prepay to reserve tokens (the words large language models process or generate), according to a person who spoke to executives.
Currently, the startup offers mainly on-demand pricing, charging developers anywhere from a few cents to $120 for every million tokens its LLMs generate. Some larger customers receive volume discounts.
Discounts for paying in advance are common in cloud computing—customers of Microsoft Azure, Google Cloud and Amazon Web Services can lower their costs by reserving server capacity ahead of time.
With more-flexible pricing, OpenAI could better compete with rival model developers, as well as the startups that aim to help developers run open-source models more cheaply, known as AI server resellers or inference providers. Their focus on cost efficiency has sometimes driven these startups to offer the same LLMs at lower and lower prices, or even below cost in some cases.
OpenAI has already introduced a way for its developers to lower costs with Batch API, an application programming interface it launched in April that provides developers with cheaper pricing if they upload model queries in bulk and are willing to wait up to 24 hours for responses. For their part, AI-server resellers such as Together AI and Anyscale say that running open-source models on their software is up to six times cheaper than using OpenAI’s models.