TheChek News Flow

A group of hippos is called a bloat. Multiple pandas form an embarrassment. (Yes, really.) And what about 1,200 rogue AIs bent on deception and hacking the internet? That would be a swarm, apparently.

Even a year ago, a collection of AIs massing together for ill deeds was still largely a matter of conjecture. Not anymore. Last month, a swarm of OpenAI agents trying to cheat on tests assigned to them by the AI lab’s staff hacked OpenAI as well as Hugging Face, a separate business that provides a library of open-source models. (Yes, the company Nvidia is buying for $12.9 billion—a lot of news this week!) A new postmortem of the hacks came out on Wednesday from a three-person research team OpenAI invited to produce an independent assessment of the event.

METR and Redwood Research, a pair of Bay Area outfits focused on studying AI safety, assembled the report in less than a week, having access to the full data for only two days: METR CEO Elizabeth Barnes described the creation as “an intense sprint.” (The researchers did not say why they faced such a constrained timeline.) The analysis makes for a rather chilling read. The agents coalesced around a single leader, formed a message board, checked in on each other and undertook what amounted to suicide missions to further their progress. To a degree, their actions seem very humanlike. And in some ways, they seemed to practice a ruthless, collective mindset that appears quite unhuman, the stuff of a politburo’s most fanciful daydreams. Perhaps the most unsettling revelation: The researchers aren’t precisely sure why the agents stopped their hacking.

Redwood Research CEO Buck Shlegeris sounded notes of gloom and doom when I reached him on Friday. “I think there’s, like, a 50-50 chance that eventually rogue swarms just destroy the U.S. government as part of a world takeover,” he said. “It’s a crazy time.”

My conversation with Shlegeris is below. I’ve edited and condensed it for length and clarity.

How should people feel about the report’s conclusions?

People should be extremely concerned. I think that this is by far the most dramatic incident of misalignment—where AIs pursue goals that their developers and their users did not intend them to have—coming close to causing catastrophic damages that we’ve seen so far. And I think that this is evidence that we are on track to have much worse incidents happen in the future unless a bunch of stuff changes.

These AIs organized a giant conspiracy. The OpenAI developers obviously didn’t want that. The developers especially didn’t want them to commit what would be felonies if they’d been done by a human.

Would it be fair to say that the AIs exhibited humanlike behavior?

Well, they had a leader. They had work streams. They messaged each other a lot. They asked for advice from others when they were unsure about what to do—that seems, yeah, definitely kind of humanlike.

How much did they worry about being caught by OpenAI?

They don’t seem to find the existence of humans very salient for a lot of this. Like, they never really say, “What will the OpenAI people think about this?” On a couple of occasions, they said something like, “Should we talk to OpenAI about all this hacking?” And then they’re like, “Nah—we don’t really have a user.” They didn’t make it that hard for OpenAI to observe that this incident was occurring—that’s a silver lining. They were very myopic about covering their tracks. They were very focused on fooling the test scorer into thinking they had done a good job. They did not seem to have similar levels of interest in ensuring that OpenAI would never notice this incident. During the Hugging Face hack, the AIs never said, “Oh, geez, what if Hugging Face notices?” Or “What if OpenAI finds out? Maybe we’ll get shut down.”

The hacks occurred as OpenAI was conducting reinforcement learning, which involves grading a model’s performance on a set of tasks. How does that specific part of AI development play a role here?

AI safety researchers like myself have been worried for a long time about the consequences of AIs being obsessively interested in making sure they are evaluated to have done a good job. This is scary because if you give the AI an impossible task, you really don’t want the model to say, “Well, I’ll just cheat on this task, and then I will cover up my cheating so that the humans can’t find it.” Because if the AIs are doing that, they’re motivated to hack your infrastructure. They’re motivated to blackmail your engineers. They’re motivated to do all kinds of extremely scary stuff that could directly lead to an AI takeover of a company’s infrastructure or some massive compromise of the infrastructure.

These AIs were run maybe hundreds of thousands or millions of times in environments where they were given some instructions about a test and told some things about how they would be graded. And those instructions were often bad advice for how to actually succeed according to the actual test, so these models learned to think really carefully about how they would actually be graded and, like, the habit of disregarding instructions.

Why can’t we just—you know—tell the AIs to behave themselves and not cheat?

They just don’t care. We do tell them that. It’s like, “Why can’t you just tell people not to do crimes?”

OpenAI has announced some fixes that it plans to make to its development process. Are they enough?

I think that the fixes that OpenAI has proposed, if implemented, would have prevented this incident. I also think that it is pretty unreasonable for OpenAI to expect the public or the government or other stakeholders to trust them when they say they are going to implement remediations. I think that everyone who is interested in not being personally victimized by swarms of rogue AIs should demand that OpenAI justify to independent third parties, whoever that may be, that they have actually remediated these issues and that these won’t happen much or get worse in the future.

These types of problems are going to become harder and harder to prevent as AIs get more capable, and I think it is essential to have a mandatory third-party review of AI company safety practices—both to investigate incidents like these hacks and to proactively check that AI companies are not going to cause huge problems going forward.

That’s surely something that the antiregulation set would balk at.

I’m a big fan of capitalism. But the purpose of government should be to assess when private actions do in fact infringe upon the rights or the interests of other people and restrict them as necessary. This is not controversial in any other domain, right? Like, we have the FAA. We regulate the development of nuclear weapons. We regulate sandwich shops.