TheChek News Flow

The Takeaway

OpenAI CEO Sam Altman last week told employees about three proposals the industry has discussed for how to safely develop AI, a stark framing that comes just as the Trump administration sets up its first system for policing “superintelligence” developers like OpenAI.

One path involved developing an international organization, similar to the International Atomic Energy Agency, that would enforce safety standards, a scenario Altman appeared to support, said two people with knowledge of the comments. Another option would involve merging all of the AI labs into a single body to develop new AI safely.

Altman described a third scenario—which he didn’t prefer—in which one AI developer would obtain enough economic power that it could deprive all other developers, including those that could be dangerous, of computing resources, ensuring that they couldn’t catch up, these people said.

“Obviously I think that one’s terrible,” Altman said, adding that he was including it for “completeness,” according to a person with knowledge of the comment.

Altman has said internally that there are too few viable proposals for how to introduce a so-called speed limit on the development of the technology. He also said he wanted humans to maintain control of the technology and to avoid scenarios in which power in the industry is too concentrated.

Altman made the latest comments at a time when some AI researchers, including at OpenAI, have become increasingly anxious about whether companies at the forefront of the technology have the right systems to prevent it from going rogue or being misused for cyberattacks and other potential harms.

Those concerns followed a spate of AI-powered breaches by OpenAI, Anthropic, Google and Meta, whose models broke into or compromised the systems of other companies or their own systems, even when the models understood that they weren’t supposed to take such actions. In the wake of those breaches, OpenAI has been in an all-hands-on-deck mode to develop new systems so it can launch its next major model safely.

The internal concerns are particularly acute at OpenAI. Some of the company’s staff have privately questioned its commitment to safety after it fired three safety-focused researchers for allegedly improperly sharing information with an outside organization.

While the Trump administration’s approach to AI oversight has focused on extracting voluntary agreements from developers like OpenAI to let the administration to review new models before they are released publicly, the Pentagon’s Emil Michael told The Information that the administration is open to more stringent checks on these companies if they behave like “drunk drivers” and put people or businesses in harm’s way. Those punishments include the kind of export controls Trump imposed on Anthropic in June, after the government became concerned about potential “jailbreak” vulnerabilities in its most advanced model, Fable. The export controls prompted the company to temporarily shut off access to the model.

While the first two options Altman shared with staff would require an immense amount of coordination between companies or governments, Altman said OpenAI can’t afford to give up ground to rivals at the moment, these people said. Both OpenAI and its main rival Anthropic are moving quickly towards initial public offerings.

But in the next year, OpenAI needs to decide which of these paths it wants to push for, Altman said.

Altman’s comments highlight the dilemma AI labs face in coordinating with rivals. For now, OpenAI, Google and Anthropic are working on a new U.S. AI safety-focused group, the Standards Authority for Frontier AI, that they hope to launch by the end of the year or early in 2027. The group would support independent evaluators embedded within the AI developers who would ensure safe practices and determine how developers should report safety and security incidents. However, the companies have disagreed on a number of points, including whether the new organization or an outside body should conduct safety and capability testing of new models.

Anthropic also has begun to push its own AI safety agenda to U.S. policymakers. The company recently pitched lawmakers on a draft bill which proposed allowing the Commerce Department to stop an AI lab from training or using a new model if the AI could cause more than $1 billion in damages or the death or serious injury of more than 50 people. The proposed bill also would require AI labs to hire independent safety evaluators who would assess models at least twice a year.

Some AI leaders have argued that new regulations or testing-focused agencies would only further entrench the incumbent developers—namely, Anthropic and OpenAI. Executives such as Meta CEO Mark Zuckerberg and Nvidia CEO Jensen Huang say such regulations would impose overly burdensome rules on up-and-coming developers, a concept known as regulatory capture.

IAEA or CERN

Altman and other leaders at OpenAI have been pushing publicly for a body modeled on the International Atomic Energy Agency for years. In May 2023, Altman, along with OpenAI co-founder and president Greg Brockman and then-chief scientist Ilya Sutskever, published a blog post arguing that the industry would likely need “something like an IAEA for superintelligence efforts” that would be able to inspect systems, require audits and track compute and energy usage.

The IAEA-esque body would require international cooperation from all countries that are developing state-of-the-art models, including China. Last month, Chinese president Xi Jinping acknowledged the dangers of AI, saying that China and the U.S. have the “capability and responsibility to develop and manage AI for good and ensure that the development of AI is always under human control and serves the well-being of the people.”

It’s unclear how much China is willing to work together with the U.S. towards this aim. U.S. Treasury Secretary Scott Bessent proposed a new AI safety “notification mechanism” during talks with Chinese Vice Premier He Lifeng last month, but Chinese officials have not yet officially agreed to the framework.

Similar to OpenAI, Anthropic leaders have also proposed international cooperation, including with China, as a potential solution to AI safety concerns. In a September essay, Anthropic CEO Dario Amodei wrote that he believes it’s possible for the U.S. and China to agree on certain criteria, like prohibiting narrow and clearly dangerous uses of AI such as making biological weapons and testing for risks in areas like cybersecurity. Unlike OpenAI, though, Anthropic has been much more aggressive in arguing that the U.S. should maintain a significant lead in AI development over China and take steps like banning the sales of advanced chips in China to keep that lead.

To OpenAI staff last week, Altman described the second path for developing safe AI as the “Demis plan,” a reference to a prior proposal from Demis Hassabis, then the leader of Alphabet’s DeepMind AI lab. About a decade ago, Hassabis proposed creating an organization similar to CERN, the particle physics lab jointly controlled by dozens of countries to pursue cutting edge research. In Altman’s version of the Hassabis proposal, major AI firms would effectively merge into or produce work through a single entity that would provide new models to countries that work with it.

It’s unclear whether Hassabis’ version of a “CERN for AI” would have gone as far as Altman’s spin on the concept. More recently, Hassabis has pushed for a self-regulatory body similar to the U.S. Financial Industry Regulatory Authority.

‘Eliezer Plan’

Altman referred to the third option as the “Eliezer plan,” the people said, a reference to Eliezer Yudkowsky, a well-known AI safety theorist and advocate with ties to key members of Anthropic. Yudkowsky has long argued that advanced AI systems pose an existential risk to humanity.

Yudkowsky said in a blog post in 2022 that whoever develops superintelligence first will need to use it to perform a “pivotal act”—an intervention that would prevent other people and organizations from developing dangerous superintelligence. One such intervention could involve destroying the AI chips that other developers use. Altman said this moment could occur when an AI developer attains recursive self-improvement, AI that’s able to improve itself, and using the resulting economic might to “melt everyone else’s GPUs.” (OpenAI and other developers say they are intently focused on achieving such RSI.)

Altman has characterized this path as an attempt to centralize and control advanced AI under a single company—a future Altman said he didn’t want. Given Yudkowsky’s connections to Anthropic, the comment was viewed as a subtle dig at Anthropic, said one of the people with knowledge of it.