TheChek News Flow

After months of build-up, developers were excited to finally see the long-awaited release of Google’s Gemini models yesterday. But their enthusiasm was tempered by the fact that the only model available right now is Gemini Nano, which you can guess means it’s small—so small it can run on Google’s Pixel phones. Developers have to wait until next week’s release of Gemini Pro, Google’s equivalent of GPT-3.5, to have a model worthy of testing, they told Jon Victor and me.

And that’s not the only reason to be cautious about coming to any conclusion about Gemini. While many developers were impressed by the multimodal capabilities Gemini showcased in a splashy video demo, replicating that sort of performance is likely to be harder than the video made it seem. (Leave aside the fact that those multimodal capabilities aren’t available yet.) In making the video, Google researchers didn’t prompt Gemini with normal speech and video visuals like hand gestures. Instead, they seem to have used a carefully crafted combination of images and text, according to this blog post. Some of the actual prompts were more specific and detailed than the dialogue in the video.

Google staffers must be feeling the pressure because they also took some additional steps to make Gemini look extra good compared to its rivals. Front and center on Gemini’s release site is a chart claiming that Gemini’s most-advanced model, Ultra, outperforms OpenAI’s GPT-4 by a large margin on a commonly-used evaluation, MMLU.

But that’s not an apples-to-apples comparison. GPT-4’s score of 86.4% was based on the industry standard for evaluation, called “5-shot.” However, Gemini Ultra’s result of 90% was based on a different method developed by Google researchers, one based on “chain-of-thought with 32 samples.” In simple terms, using this approach means that, for the same question, Gemini Ultra generates 32 answers and the reasoning for those answers. Then, the model chooses the most-commonly provided answer as its final one.

It could be that this new approach allows Gemini to “reason” through questions better. But using the industry standard—5-shot MMLU—for both models, GPT-4’s 86.4% outranked Gemini Ultra’s 83.7%.

Still, there’s been lots of discussion around how these model evaluations can be hacked, so we’ll be waiting for the release of Gemini Pro and Ultra to make our final verdict. Until then, take things with a grain of salt.—Stephanie Palazzolo and Jon Victor

Here’s what else is going on…

Watch Out for Model Hijacking

Over the past year, we’ve seen plenty of examples of large language models spitting out worrisome content, from encouraging users to harm themselves to providing them with instructions on how to build bombs. (Just yesterday, we featured a new security threat to LLMs.) These incidents have caught the eye of security experts and raised concerns around the safety of generative AI models.

As headline-grabbing as these examples are, though, there’s a much less sexy but equally dangerous security threat in AI: supply chain attacks. In plain English, supply chain attacks refer to a type of cyber attack that targets a vendor who sells services or software used by other companies to build their products.

Cybersecurity startup Legit Security recently uncovered a supply chain weakness in the popular open-source model repository Hugging Face, highlighting some of the often-overlooked downsides of using open-source AI.

Commonly, developers will use open-source models or datasets from the Hugging Face platform when building AI-powered products. When the owners of these models or datasets update or rename them, any products using the models or datasets automatically begin using the updated versions to prevent disruptions, said Liav Caspi, co-founder of Legit Security.

The problem arises when a developer registers a new model or dataset under the old name. When that happens, products can start accidentally using that model or dataset instead, Caspi explained. That allows attackers to create new malicious models that borrow names from legitimate ones in order to feed poisoned data or malware into users’ apps, he said. Legit Security used the hack to prove that hundreds of thousands of developer apps that relied on the “ai-forever” models could quickly be corrupted, he said.

Legit Security dubbed this bug “AIJacking.” And when Caspi’s team alerted Hugging Face to the bug, Hugging Face said it had a mechanism that automatically retired old names of the most popular models and datasets once developers renamed them. That leaves Hugging Face to manually retire less-popular models, which Caspi worried isn’t a scalable solution. A spokesperson for Hugging Face declined to comment.

The incident shows that while regulators and doomers worry about AI ending humanity, or terrorists creating bioweapons with open-source models, a variety of more immediate security threats looms right now.

Developers can guard against AIJacking attacks by referencing a specific version of a model hosted by Hugging Face, and preventing their apps from automatically executing code from third-party vendors like Hugging Face, Caspi said.

The AIJacking flaw is also a stark reminder of the dangers of building software on top of young and relatively untested startups, as we saw with the controversy surrounding OpenAI in recent weeks. That debacle prompted a number of customers to consider switching to OpenAI models offered by a tech veteran like Microsoft instead.