TheChek News Flow

Thankfully for OpenAI, the conference ended before Apple filed its blockbuster lawsuit accusing key OpenAI device designers and engineers of stealing trade secrets when they left Apple. And for those of you wondering, OpenAI’s booth at the event was popular while Anthropic didn’t have much of a presence. Five themes dominated the conference:

1. Researchers get practical. Researchers at ICML were focused on practical questions like how to lower the costs of running and training large language models. That’s a shift from the mood of those who attended the Neural Information Processing Systems conference held seven months ago, where some researchers called for a total overhaul of the field. Up-and-coming AI labs like Ilya Sutskever’s Safe Superintelligence and Jerry Tworek’s Core Automation, which back then had promised to upend the architecture underlying today’s most popular AI products, haven’t released anything publicly yet.

Instead, multiple papers presented at ICML involved improving quantization, a technique that makes models cheaper and faster by storing their weights, or “settings,” less precisely, and key-value caching, which helps a model remember information from prior calculations it made so it doesn’t need to repeat the work. Both of those methods lower the costs of running models, which is becoming increasingly important as the size of models grows, as measured by how many parameters they have. (Clearly, OpenAI and SpaceXAI are taking advantage of such research, as we’ve recently reported.)

2. Is continual learning overrated? At the end of last year, continual learning, the idea that AI models should continue to learn even after they’re trained, was all the rage. Though many researchers still highlight the promise of continual learning, they’re also quick to admit that today’s models are clearly good enough that businesses and developers will fork over billions of dollars to OpenAI and Anthropic to use them. As one Thinking Machines Lab researcher at ICML told me: “Continual learning is like a luxury—it would be good to have but labs are clearly doing well enough without it.”

Meanwhile, OpenAI researcher Noam Brown said in a panel discussion that he’s still “scaling-pilled,” meaning that he believes that simply scaling up the amount of compute and data that existing models are trained on will continue leading to big gains in model performance. Brown added that reaching a point where AI is good enough to conduct AI research could help to take AI development the rest of the way.

3. The promise of diffusion language models. Even if researchers weren’t as enthusiastic about new architectures and approaches like continual learning as in the past, one new type of model that still seemed to hold their interest was diffusion language models, which appeared in a number of papers.

In contrast with autoregressive language models, which writes responses from left to right and underlies today’s most popular products like ChatGPT and Claude, diffusion language models work by giving a complete answer that looks messy and seemingly random, and then repeatedly refine it into a final response.

Diffusion has traditionally been used in image generation models but more recently has been applied to language-generating AI by startups like Inception Labs. One of its biggest advantages is that it’s much faster than autoregressive language models, especially in the case of generating longer responses.

Because diffusion is so different from autoregressive models, though, it’s unlikely that the big labs will upend their current methods of developing models in favor of it—and, again, they seem to be doing economically well enough without it. But if diffusion language models become significantly better, faster or cheaper than autoregressive ones, all bets are off.

4. “Unlearning” is just as important as learning. Lots of papers at ICML focused on the concept of “unlearning,” which is teaching a model how to forget information. Unlearning is important in cases where a model should “forget” information it was trained on because, for instance, that data contains sensitive or personal information or because there is newer, more updated information it should use instead.

However, unlearning is difficult because models don’t store their training data like a cell in an Excel spreadsheet that you can delete. Instead, that data is spread out across billions of weights, or the “settings” that control how a model responds. That means that it’s difficult to delete one piece of information without affecting lots of other parts of the model. Plus, researchers already have enough problems with “catastrophic forgetting,” a tendency of models to forget all old information when they’re trained on new data.

5. The Trump administration’s shadow looms. During an OpenAI panel discussion about safety, some ICML attendees asked OpenAI staffers how they were thinking about future model releases given the widespread cybersecurity concerns that have delayed or otherwise affected recent AI releases. (Just last week, OpenAI finally released its flagship GPT-5.6 family of models publicly after the Trump administration initially asked it to release the models in a limited fashion.)

The safety staffers weren’t shy in sharing how difficult it is to balance the benefits of having models out there which can help identify and fix vulnerabilities with the potential harms of those same models being able to exploit vulnerabilities. Dylan Scandinaro, OpenAI’s head of preparedness, said that it’s frustrating that models like Anthropic’s Fable 5 can’t answer a question as simple as “check this code for any bugs,” especially when the pace of AI development isn’t slowing down and attackers can ostensibly use open-source and other models to exploit vulnerabilities.