Meta Develops AI Search Engine to Lessen Reliance on Google, Microsoft
The Takeaway
• Meta is crawling the web so its AI chatbot can answer questions on current events
• The work aims to avoid relying on Google and Bing for real-time information
• Meta has been working on web crawling for at least eight months
As Meta Platforms tries to keep up with OpenAI in developing artificial intelligence, the Facebook owner is working on a search engine that crawls the web to provide conversational answers about current events to people using its Meta AI chatbot.
In doing so, Meta hopes to lower its reliance on Google Search and Microsoft’s Bing, which currently provide information about news, sports and stocks to people using Meta AI, according to a person who has spoken with the search engine team. It could also give Meta a backup option if Google or Microsoft withdrew from these arrangements, according to a person who has been involved with the strategy.
The efforts show the lengths to which Meta CEO Mark Zuckerberg is going to reduce Meta’s need for other major technology providers. Zuckerberg has been stung by Meta’s dependence on another big tech firm, Apple, which several years ago made it harder for Meta to generate ad revenue through its iPhone apps.
In another potential step toward divesting from Google and Bing, Meta recently struck a deal with news agency Reuters to help Meta AI answer questions about current events and news.
Spokespeople for Meta and Google declined to comment. A Microsoft spokesperson did not respond to a request for comment.
It isn’t clear whether Meta pays Google or Microsoft for powering answers to questions to its chatbot. Zuckerberg said in an interview in April that “there’s not a ton of money flowing either way” between Meta and Google, without elaborating.
Whatever the situation now, Meta may be concerned about what could happen in the future. People are increasingly using AI apps such as ChatGPT in lieu of traditional search engines such as Google. To handle topical queries that require web access, OpenAI relies on Bing, whose owner is OpenAI’s biggest outside shareholder.
Meta AI is the centerpiece of Meta’s strategy to capitalize on the boom in conversational AI and increase the time people spend on its apps. Meta hopes to eventually develop AI to power agents that can perform complex tasks without humans, such as developing software or helping businesses advertise on Meta’s apps. The chatbot is prominently positioned on Facebook, Instagram, WhatsApp and Messenger.
Zuckerberg said in August that Meta AI has more than 185 million weekly active users. That suggests it is catching up to ChatGPT, which OpenAI said this month has more than 250 million weekly active users. People likely use the two chatbots for different purposes, since Meta’s is built into its social media apps, while software engineers frequently use ChatGPT to generate or vet code.
Meta AI’s search engine team—which has been led by senior engineering manager Xueyuan Su—has been working on accessing websites and then organizing their pages into databases, a process known as web indexing, for at least eight months, according to the person who has spoken with the team. Meta AI will eventually be able to access those web indices to answer relevant questions. The team’s existence hasn’t been previously reported.
Meta’s web-crawler technology has been publicly disclosed since at least July. Meta says it uses a crawler bot “for use cases such as training AI models or improving products by indexing content directly.” Some websites, including The New York Times, have blocked the bot from accessing their content.