TheChek News Flow

Recursive Self-Improvement Won’t Be Winner-Takes-All, Say Founders of Ricursive

Recursive self-improvement, the idea that AI models could get so good at research and development that they train themselves, without a need for human researchers, has captivated Silicon Valley. Some in the industry have suggested that the first company to achieve RSI will gain all the benefits: after all, once one company manages to cross that threshold, how could competitors still at least partly reliant on slow human labor ever hope to catch up?

But just because the pace of RSI will be exponentially fast doesn’t mean it’s winner-takes-all, argued Anna Goldie and Azalia Mirhoseini on the latest episode of The Information’s AI Deep Dive. They are the co-founders of Ricursive Intelligence, a year-old startup developing AI systems for designing AI chips. They previously worked at Anthropic and Google, where they led work on AlphaChip, an AI system for chip design.

For one thing, an AI company that focuses only on improving its AI models could be overtaken by companies like Ricursive that are co-designing both the models and the chips that power them, argued Mirhoseini. “Imagine you could train your next model by an order of magnitude cheaper than, like, the current frontier lab,” she said. By getting ten times as much training out of the same chips, that would allow a company to “leapfrog a generation or two of model development.”

And even if one company’s language models zoom ahead of the pack, there will be room for an “ecosystem” of AI companies, said Goldie. For example, even a language model that is superhuman at research might still need a specialized AI system like the one Ricursive is building if it wants to design its own AI chips, she argued.

We also discussed the hardest parts of making chips, why custom chips tend to be designed for running existing AI models rather than training new ones and the safety concerns that RSI raises.

The following transcript of the conversation has been lightly edited for clarity.

Drew: Welcome back to the Information’s AI Deep Dive. On this show, we get to the bottom of the hardest technical problems in AI. I’m your host Rocket Drew, and my guests today are Anna Goldie and Azalia Mirhoseini, the co-founders of Ricursive Intelligence. The pair have worked together for the past 10 years. They started at Google together, where they went on to work on AlphaChip, an AI system for designing AI chips. They left and went to Anthropic on the same day. They went back to Google on the same day, and of course, they started Ricursive Intelligence on the same day. Now they’re on a podcast together on the same day. So thanks so much for coming on the show. I’m so glad you’re here.

Goldie: Yeah, us too. Yeah, thanks for having us.

Drew: Yeah, of course, of course. So the topic of our conversation today is recursive self-improvement, which is all the talk of the town in Silicon Valley and San Francisco especially, these days. I was hoping you could start by explaining to us what is recursive self-improvement. What do people mean by that? And then we can get into your unique take on recursive self-improvement, which involves AI chips.

Goldie: So recursive self improvement is AI improving itself, but from our perspective, we think AI is the entire system: model, software, hardware, and not just at the single chip level, but at the multi-chip, at the rack, at the data center level. So co-optimization across all of these different layers.

Mirhoseini: Yeah, and it becomes recursive when AI itself is at the core of all the optimizations and like creations of new generations of this and entire stack—again, from model to software to hardware and so on.

Drew: Okay, so sometimes people talk about recursive self-improvement as shorthand for automating the work of the research scientists at frontier labs like OpenAI and Anthropic. That you’re automating research and development itself. You’re saying that’s thinking almost too small. You have to automate—

Goldie: Too shallow, actually.

Drew: Too shallow, okay. You have to automate everything else that’s—

Mirhoseini: Full stack.

Drew: Full stack.

Goldie: Exactly. Like it’s like the human brain is not just the software layer: there’s a physical substrate.

Drew: So there’s a few aspects of this that I’d like to get into, including what is challenging about recursive self-improvement and what are the hardest parts to automate here. But I think at a high level, this loop makes sense. That you have AIs that design better chips, and those chips then help design better AIs. I want to start with the second leg there, the chips that design better AIs, and then we can return to the other direction. What does it mean for some chips to be better than others for designing—for training AI systems, or even running AI systems? I think that might not be obvious to some people.

Mirhoseini: For example, you could have a training chip right now, you have a bunch of GPUs or TPUs and so on, training your current AI models and you are at say 30% MFU—model flop utilization. That means, like, even within the realms of what the chip that you have today, you’re only using 30% of this chip kind of offering. If you 3X that, if you go closer to 100%, that means that you can potentially train this model three times faster. That means your model gets to this newer generation or the next iteration of it three times faster. So basically, you’re optimizing or you’re speeding up the inner loop of evolution of AI models, and that’s how, that’s one way to look at this recursive self-improvement with chips at the core of it.

Drew: I see, I see. And which steps in the process of creating chips have the most potential for automation right now? My understanding is creating a new chip. There’s a lot of steps that go into that, and a lot of them are slow and involve interacting with the physical world. What can be automated, and what’s harder to automate?

Goldie: I mean we think all of it can be automated.

Drew: Okay.

Goldie: But what are the highest ROI, what are the pieces that really are slowing down this recursive loop? So the two long poles right now: physical design and design verification. So physical design is how do you generate the physical layout of this computer chip. How do you place these hundreds of millions or even billions of logic gates onto this multi-layered wafer, subject to multiple different physical constraints? And this, as it stands, it can take months, even more than a year, for teams of tens or hundreds or even thousands of human experts using very expensive tools, and we don’t think that’s necessary. And we also think that the performance that is possible to achieve is, you know, we’re leaving performance on the table right now, given these slow, suboptimal, human-in-the-loop methods.

And then the other long pole is design verification. So how do you make sure that the chip is actually correct? Because in software, let’s say you have a bug, it can be very, like, damaging, but you can fix it post facto. In a chip, you know, it’s much harder to fix a chip after it’s been taped out. You might lose a hundred million dollar mask. So, design verification is also something very slow, very suboptimal. There’s kind of two main ways that it can be done, and right now it’s typically done using dynamic verification, which is: visit every possible state of the chip in every possible sequence and make sure that you’re never in a state that you didn’t expect. And that’s basically impossible given the scale of current chips.

Drew: How many spaces is that? Like order of magnitude?

Goldie: It’s very extreme, like more than the number of atoms in the universe.

Drew: Oh yeah, no problem.

Goldie: So people typically they spend like over a year, and then they just hope they’re like, “Okay, we didn’t find any bugs, so we hope that we’re good and we’re gonna tape this out.”

Drew: We just sampled enough.

Goldie: And the other path would be formal verification, which would be mathematically proving that the chip is correct. Historically, that’s been intractable, but we think there’s hope.

Drew: Okay. What does that look like? What makes it tractable now?

Mirhoseini: Um. So basically, I think that’s—

Goldie: Maybe that’s the secret sauce.

Mirhoseini: That’s the secret sauce. That’s what we’re building. But at a high level, we are designing algorithms that scale with compute, and they’re intelligent, meaning that they become better. Again, there is that recursive self-improvement. They become better as we solve more instances of the problem, so we are turning this entire kind of like series of combinatorial optimization problems at massive scale into algorithms that are tractable, that scale with compute, and they’re very efficient, and they can self-improve over time.

Drew: For both of these problems, for design and for verification.

Mirhoseini: Yes,

Drew: I see.

Goldie: Physical design and design verification, yeah.

Drew: Physical design and design verification.

Goldie: And there are other interesting problems too. So I think those are the things that make it take a very long time to tape out a chip. But there are other aspects of chip design that disproportionately impact the performance of the chip. So for example, the computer architecture design, that’s where you can get—you know, at physical design stage, maybe if you do a really good job with layout, you could get a 20% improvement in power efficiency. If you did a really good job with computer architecture, customizing the chip to the workload, you could get a 20X or more. But you have to be able to quickly tape it out for it to be a relevant chip.

Drew: I see. Which of these stages involves the most interaction with the physical world. Maybe in the worst case, like you said, you tape out a chip, and then something is wrong, and you have to go back and tape it out again. This can be very painful.

Goldie: That’s really bad. Very expensive, and very slow too.

Mirhoseini: Towards the back end of chip design flow, you have more and more interactions with the physical world, and that constitutes physical design, which has to do with place and route of placing and routing all the components of the chip, and then sign off. And those are the stages right before you send this chip design to the fab for fabrication. So those are the stages you have to be maximally aware and consider all of the physics that matter here.

Goldie: I think even after fabrication, then you have to do bring-up, you have to do packaging, you have to do all sorts of physical testing afterwards.

Drew: You can imagine that those physical tasks, as automated as the other steps get, will continue to be bottlenecks on the speed of bringing new chips online. How fast could you imagine this whole cycle getting end to end, and how long does it take right now?

Mirhoseini: So right now, it depends on the size of the chip. It could take multiple years to bring out an entirely new, like giant and very powerful chip from the design all the way to tape out and to something that’s out there in the data centers.

Right now, the design part is takes the vast majority of this time when you go from the spec all the way to sign-off, and tape out does take, manufacturing does take time, but still it’s shorter relatively to the design time, so at recursive right now our focus is drastically reducing the design time. Basically, we are turning this problem into a problem that scales with compute, and that becomes a knob. We throw more compute at the problem, and we reduce the the time to design, and that can be something that we can control.

On the manufacturing side, there are ways that we could be helpful to the manufacturing and to the fabs as well. They are taking a lot of approaches themselves right now, because there are a lot of complexities, a lot of data-driven decisions that can be carried by intelligent systems, and that would significantly reduce the runtime of the fabs as well. So this trend is gonna have a kind of a flywheel effect. If we drastically shorten the design time, the fab becomes—shortening that becomes more important, and there’s more incentive for fabs to do so.

Drew: I see. And the techniques that you’re pursuing, are they general purpose in the sense that they could aid in the design of arbitrary chips, or what are the kind of chips that you’re targeting? Even, maybe one way to break this down is: could your techniques be used just for chips that are designed for inference or running existing models, or also for training? Is there a trade-off there?

Goldie: I think certainly on the back end, it’s completely orthogonal. Like the physical design part is the same, whether it’s a training chip, an inference chip, what the workload is. I think as you get closer to the beginning of the pipeline and the front end, then it’s more like you’re doing customization for the workload, so of course it matters what the workload is. But we’re generating, we’re designing a general pipeline for this. Training and inference.

Drew: Training and inference.

Goldie: Yeah.

Drew: My sense is that a lot of the existing custom chips are optimized for inference, not for training. What’s the reason for that, and why does that change in your case?

Goldie: It’s easier. Go ahead.

Mirhoseini: It’s definitely easier to make an inference chip. One reason is just the workload is easier, and also you don’t have to have a massive rack to begin with to create an inference chip because an inference chip, like even a single chip, can run a model that’s very useful. Whereas for training, due to backpropagation and like these massive amount of weights and optimizers, it’s all of these that you carry, you have to have this massive number of chips interconnected in ways that allows you to efficiently train your model. So it’s like a much more complex thing. So that’s one side.

The other side is just the demand. There are very fewer companies that are training frontier level models, versus pretty much every company, every industry needs inference and needs to use these models that are trained by a few select providers. So inference, there’s a lot more demand for inference.

Goldie: The other part is like the software layer is more challenging for training chips.

Drew: Why is that?

Goldie: Because, like for inference, you just serve a particular workload. You don’t need the complexity of like forward pass, backward pass, parameter update, all of that.

Drew: Yeah, yeah, that makes sense. The complexity that gets added when you need multiple chips to communicate with each other during training, that’s also an issue when it comes to designing the chips for you two, but AI maybe plays a role in helping with that, or?

Mirhoseini: Absolutely. So we at Recursive, we really, the core of everything we do is co-design, and that co-design is not on just the chiplet or the chip, the device that we’re building, but it’s the entire kind of the the rack and how these chips are connected with each other, how they’re designed with respect to that configuration and so on.

And the way we approach this problem is, again, this is like a very complex combinatorial optimization problem, and we need to use it—we need to treat it as this like math objective. Then there are ways that we can solve them using the tools that we have in math. So that is our approach, our thesis about how we approach these problems.

And if you think about it that way, a lot of these optimizations become much more doable, and the part on the human experts that work on these problems is about the creativity or bringing in the ingredients that we can use for co-design rather than doing the optimization itself. The optimization itself can be done by AI and other accelerated methods that we have.

Goldie: Exactly, like what’s the search space, versus like today, humans are like, they have spreadsheets and they’re using that to design very very high impact chips.

Drew: Sure, sure, yeah. There’s a wide space of possibilities here. You’re jointly optimizing this, like, very complex mathematical object, but AI should be able to do a better job than what we’re doing today manually.

Goldie: Yeah.

Mirhoseini: An opportunity here is that we can measure things for chips. There are verifiable rewards. Like we can measure performance, power, area. We can optimize things for different workload scenarios. Like things are measurable in a very objective way, and that’s very helpful.

Drew: And measurable in simulations, so you can run a very tight feedback loop before ever touching the world, ideally.

Goldie: Exactly.

Drew: Jensen at Nvidia, of course, also talks a lot about “extreme co-design,” he loves this phrase.

Goldie: Oh, we like it too.

Drew: You do. When you say it, do you mean the same thing as him, or is there a distinction in the approach?

Goldie: What does he mean exactly by it?

Drew: I think the same thing, effectively? That they’re jointly designing the chips and the racks, but maybe not that they’re co-designing the models and the chips. Though, of course, I’m sure they are designing their chips with an eye towards where the frontier is moving on the model layer.

Goldie: I think that yeah, we mean the same then.

Mirhoseini: We mean the same. I think the focus in our company also is like really, really accelerating the chip design cycle. So this co-design become even more meaningful because if we can customize the chip very quickly to the models that are being developed and evolve so quickly, then we can take this co-design chip to something that’s out there and can be used readily much faster. So we can customize even further because we can adapt rapidly to the changes in the model.

Drew: Just to return to the idea of recursive self-improvement, I wanted to understand… It’s conceivable that you have an AI that designs lots of chips for inference, and they could be specialized in different ways. But that doesn’t lead to a self-fulfilling cycle because these are chips that are optimized for running AI models for a certain purpose, not training them. Even if you’re designing chips that are optimized for training a model, it could be for training narrow weather models, say, and then you don’t get the feedback. It’s not recursive. So it seems to me that someone has to be working on chips to design models that are used to design chips that are used to train models and so on.

Mirhoseini: I mean definitely that’s what we are very excited about. But just going back to just like even the inference chips, our employees and our team and pretty much everyone in every frontier lab, they’re using AI, like this inference engine, to come up with better ideas, to improve their algorithms, to design new model architectures, new training paradigms. So it is true that there is these chips and these AIs, even at inference, are directly helping the next generation, so this self-improvement is already there. But if we kind of bring it back to the model that is designed and customized to do this self-improvement loop and make the chip for that, of course, the acceleration or that inner loop is even faster.

Drew: Yeah, that’s a good point. That you need inference chips to sort of accelerate your current employees. I guess the question is, in a world where there are no human employees, or there are minimally humans in the loop, the question is what purpose they serve then. But presumably you’re still doing a lot of inference in that world too.

Goldie: I think maybe we’re, yeah we’re not there yet.

Drew: Yeah.

Goldie: I would say our employees are much smarter than the models still.

Drew: That’s reassuring.

Goldie: But eventually, yeah. I think that what we’ve seen is that very brilliant people, like, amplified with these tools can, yeah, make a lot more progress. This inference feeds back.

Drew: So it’s your company that is going to own that whole loop, that whole cycle. But maybe customers come in who aren’t in the recursive self-improvement business themselves. They want to run existing models or they want to train models. But maybe those models are kind of narrow. It’s Recursive Intelligence that will own that sort of end-to-end process.

Goldie: I mean, so our we’re an end-to-end like AI provider, like chip design service. So we can go from like a workload, like a model, we can design an architecture for that and we can do all of the backend implementation all the way to DDS. And we can also help with co-design of the model to make it suit even better to the chip that we can design for them.

So that’s, I mean maybe in terms of like recursive self-improvement, like another like part of our perspective here is that there’s this deadlock right now, right, where chips are designed for current models, and models are designed—current models are designed for current chips and we kind of can’t break out of that because each of these edges is too slow.

Drew: Yeah.

Goldie: So we kind of, even if it’s like we’re all part of this ecosystem, if we can enable like you know new frontier labs that have architectures that don’t happen to run well on current hardware, but which could run well on a chip that we could help them design quickly, then you know we can kind of together.

Drew: Yeah, you could totally imagine that we’re stuck in some kind of equilibrium, some local minimum.

Goldie: Yes, exactly.

Drew: One question I would have is, do you have a bet on sort of how far on the tech tree do we have to backtrack, both for hardware and for architecture? Like, do you imagine sort of exploring non-transformer architectures, like going that far back?

Goldie: Maybe, because I remember actually I met Jensen once, and he was talking about how at Google, like, you know, they let a thousand flowers bloom, and then one of them was transformer, and for him it was very immediately obvious that this model would parallelize as well on GPU, and it’s true, right? I think transformer is beautiful. In many ways, but one of the, I think the most powerful thing about it is how beautifully suited it is to GPUs.

Drew: But you could imagine that if there were other hardware available, and other architectures available, then we can find a better combination.

Goldie: It’s probably a global optimum that’s very far from where we are.

Mirhoseini: We’re already seeing other architectures, including one from my lab and Chris Re’s lab at Stanford, so there are newer architectures that are coming. A problem with taking those ideas into something that’s at the frontier is again the chips, because if your idea is like, I don’t know, at 2X better than some existing idea, but the chip is like 4X worse for that, then you’re at a disadvantage. So I think this kind of like accelerated chip design and what we are doing at Ricursive, which being a chip design platform, we can enable many different ideas to come to fruitation at scale, and that’s our goal.

And this is not just on the capability side. This could be models that are designed with safety constraints in mind, designed with other kind of criteria that currently we don’t have, but we can enable the physical substrates for them.

Drew: To clarify, your AI systems for chip design: are they LLMs, are they language models, or they’re more specialized systems than that?

Goldie: They’re both. Both are true. So LLMs and other types of AI methods that are not LLMs.

Drew: I see, but you are training your own LLMs currently, or okay. And those LLMs help with chip design. Okay, in what ways? Like, what does that look like? As an employee, you’re trying to design a chip, when do you turn to a specialized system, and when are LLMs the right tool for the job?

Goldie: I mean, at a very high level, you could say, like, the things that humans are good at are kind of like the things that LLMs are good at, like language related tasks, code generation, those kind of things, but massive combinatorial optimizations where you’re placing hundreds of millions of components and routing them, that’s not something that humans are good at. That’s not something LLMs are particularly good at, at least in our experience.

Drew: Yeah, it doesn’t seem like the kind of task.

Goldie: Yeah, it’d be very slow and very expensive.

Drew: I guess there’s a question here. Certainly, there’s an advantage to co-designing chips with the AIs. If you sort of had all the time in the world, like you said, we could get to a global optimum. But designing chips seems really hard, and I think a lot of companies have the sense that they’re already quite close to whatever this RSI loop might look like. Allocating any resources towards chip design seems like maybe it’s taking resources away from making AI’s better research engineers. Why should the sort of optimal policy here involve chip design at all? Chip design just seems so hard based on everything that we’ve covered.

Goldie: I mean, we can help them, so doesn’t have to be that hard.

Drew: Sure.

Goldie: But I mean, scaling laws are real. Like, if you, as you have more compute to apply to a problem, you get more intelligence, and if you have a better suited chip for the model workload, you can get very substantial compute efficiency multipliers. So that’s just pushing you out on that scaling law curve. Like, of course people should care about that if they care about model capability. I guess our perspective, and we think it’s the perspective of any AI frontier researcher, it’s cost versus capability, right? Because you can always spend more if you want to be on the Pareto optimal curve of that.

Drew: But do you imagine pushing out that curve in both directions, like price-performance sort of gets more favorable at every point.

Mirhoseini: Yeah, it compounds.

Drew: Can you say more about that?

Mirhoseini: So you can do models improving model shifting this cost capability just on the model side for a fixed hardware, or you can imagine hardware adapting to these models as well, the benefit of model improvement and hardware improvement in a lot of ways can compound, and you have even better gains, compute efficiency gains.

And it is like that’s what we are seeing in frontier labs, the ones who can afford it, they all have their own chip, custom chip teams, and they want to make the chip more and more efficient for their workloads, or at least have one kind of set of bets on that. What we are doing here is to be that platform for any company who wants, who has a workload at sufficient enough scale, whether it’s training or tests or inference, to be able to leverage that customization on the chip side.

Goldie: It’s like compute is a bottleneck. There’s limited fab capacity. If you could get more per wafer, like that, I mean that’s pretty much everything.

Drew: To press the idea a little more, I guess it’s possible that there’s a global optimum that’s much more efficient. But like, how much more efficient? Like, how much overhead is there really? It seems possible to me that, you know, maybe the current transformer architecture, the current chips, are say within 50% of the optimum—No, you’re like that’s way off.

Goldie: I feel like we have an existence proof here. Human brains are way more data and power efficient than current models. And they’re probably, I’m sure these aren’t optimal, right? No way.

Drew: Yeah, it definitely was not what was being optimized for during evolution. But take it a little further. I guess like even if we’re much further than within 50% of the optimum. You still might think that, say, OpenAI is very close. They already are designing their own chip, but even if they weren’t, even if they were just reliant on GPUs, that they’re very close to starting some kind of RSI loop. And if that’s the case, then it’s sort of winner takes all. Whoever gets there first accelerates, and it will take a long time to catch up to them.

Goldie: It’s all about the chips, right? Dario himself was saying, like in this race with China, the number one thing we can do is restrict access to chips.

Mirhoseini: I think it’s true that once you get to—and everything is like, it’s not a binary, even the recursive self-improvement. It’s not binary. It’s like there are different strengths of that that you can have. But imagine you could train your next model by an order of magnitude cheaper than, like, the current frontier lab. That means you can get there if you spend that whole value, the cost that they spend, you can get, you can maybe leapfrog a generation or two of model development because you had much more efficient ways of training this model. So it is, this kind of winner takes all doesn’t—pretty much doesn’t exist, as long as there are rooms for optimization, which we believe there is.

And just just to give you an example, one data point, we looked into this metric that we define: intelligence per watt on models, like smaller models, which we define as models with less than 20B active parameters, and these models over the period of 16 months improved by a factor of 18X in intelligence per watt, and 5.9X of it was coming from hardware improvement. So if we think about this, it means that—and that was just like that, at that time, there was no kind of pretty much very little attention to local models and improving hardware for local models. So that means, that’s just one indication of how much room we have for improvement on this cost of intelligence.

Drew: When was that that you did that analysis, in what time period?

Mirhoseini: 2023 to 2025?

Drew: I see, I see. Okay, looking at the hardware improvements that were coming online during that period.

Goldie: Model and hardware. Maybe the other piece is this: like this is exponential, right? It’s compounding, so the exponent matters for an exponential, a whole lot. So, if hardware can contribute a substantial amount to that, of course, it matters.

Drew: Because it’s exponential, I mean, I think that’s the reason people think that it is winner takes all, is whoever starts that process, starts to pull away.

Goldie: But whoever wins maybe has the hardware too.

Drew: I think that there’s a lot of reasons to think that whoever gets to recursive self-improvement first will start moving very quickly, and the reason to think it’s winner takes all is if they’re moving very quickly, someone will have to catch up to them. It sounds like you say you could catch up at some point because you’re co-designing the hardware more aggressively, you have more degrees of freedom to explore other architectures. Is that how you’re imagining it?

Goldie: I don’t think that’s how we’re thinking about it.

Goldie: Like we’re happy to be helpful to OpenAI or Anthropic, right? Like we’re not. Also, like we see very like many opportunities to have beneficial applications for AI and chips together, and we like to amplify those.

Drew: I think when I was asking before about who is owning this RSI loop, my impression was that you were like, we are owning the RSI loop, like we’re going to be doing that.

Goldie: I mean, I think it’s like it can be an ecosystem, right? As I was saying, let’s say we help another frontier lab, you know, design a chip for their model, and then they quickly, more quickly, train their model, and like, or let’s say we, you use Claude or something as a coding agent, or you know, I mean, there’s like lots of ways—It’s not just it has to be one company. It can be a system of companies, an ecosystem.

Mirhoseini: Yeah, I think, and maybe the way we think about it is to optimize the cost of intelligence, or like to get a lot for any dollars spent here, we want to create a lot more intelligent and much more intelligent systems, and that has to do, and that creates a lot of other positive impact in the world. That includes with the carbon footprint of the model, making the energy usage lower, kind of makes it that we are not as much bottlenecked as we are by the wafer and like fab capacity. So that is the kind of thing that we think this kind of like co-design can enable us.

And we can help other other entities like frontier labs, hyperscalers, chip designers, other new and upcoming AI providers, like all this physical intelligence and so on. We can help all of those achieve this kind of like efficiency or even enablement of their applications that otherwise wouldn’t have been possible with the chips that we make for them, and as part of it is like they can have they can train their models or they can get to the next generation faster and cheaper, and we think that’s kind of a positive impact that we want to have in this ecosystem.

Drew: Currently, it makes a lot of sense to me that you’re like, well, OpenAI could be a customer. They could use our software, our AI systems to design Jalapeno 2.0 or what have you. Like the question is like, OpenAI is starting very far ahead. Potentially, they are close to RSI, a lot of people in the industry think that we are close to RSI. So once they start, the pace of progress maybe goes even much faster. It goes exponential, like you said. Setting aside the question of whether anyone could catch up at that point—maybe yes. It seems like that process people think could arrive at superintelligence, meaning systems that are wildly superhuman, at all kinds of knowledge work, anything that involves using a computer.

Maybe you don’t think that’s possible. Maybe you think there are other bottlenecks that will kick in first, and the hardware will kind of be this really limiting constraint, I’d be curious if that’s the case.

But if not, OpenAI is going to end up in a world where they have this superhuman researcher, and for every hour of work that an OpenAI research scientist is doing, an overwhelming amount of labor is being done by these superintelligences. And my question is, what does chip design look like in that world?

Mirhoseini: Yeah, I think we certainly don’t think of it as, like, such a binary thing, because even in an exponential world, the rate that your exponent matters, and while you might be at the frontier, others…

And humans are, it’s very interesting, like a lot of like innovations in these labs happen kind of around the same time or within a few months of each other. Although there is no, like, these labs are completely, there are silos and there are walls in between them. So humans come with, come up, there are a lot of smart humans all around the world, they come up with new ideas, and then the rate, this exponent can vary, and as you know, when things are exponential, that matters a lot.

So we don’t think that it’s going to be like a binary, like one company gets to RSI and that’s it. And we think very positively about how the future looks like. There’s going to be an abundance of intelligent, and that’s going to help with a lot of positive applications. And we want to enable that by basically maximizing the amount of intelligence that we can get per cost, per dollar cost, per energy, and so on that we spend to create that intelligence system. So that’s how we think about this. It’s kind of, like, unlikely that it’s just like one company gets the RSI and it just feeds itself and then that’s it. Everyone else can just sit and watch. It’s very unlikely that that happens.

Drew: Yeah, I guess you could also say that even if one company does get to system, a superintelligence that’s a better researcher than any human researcher, like you said earlier, even the best human researchers—

Goldie: They’re not good at this.

Drew: They’re not good, yeah. There’s like a lot of headroom above—

Goldie: It’s brutal. It’s like such a painful task. You have to place these 100 million like Nands and Or gates, and then actually wire them.

Drew: Sounds really fun.

Goldie: No, it’s horrible. And it’s even like humans using the software, they hate it, actually.

Drew: Really?

Goldie: It’s the most annoying part of chip design.

Drew: Yeah, so you can imagine, like even an AI researcher that’s very good still having some need for AI systems to help it with chip design.

Goldie: Or at least LLMs, like, yeah. I agree with you.

Drew: You could have LLMs that have need for some more specialized system, you mean?

Goldie: Yeah, exactly.

Drew: I see. I guess. Recursive self-improvement has been in the news a lot lately because people are voicing a variety of concerns about this idea in the sense that the pace of progress is going too fast or very soon could go too fast. We recently saw Jacob Coxon, the Anthropic researcher, resign over these concerns that AI could soon do his job, and he was kind of freaked out about that. And then we’ve seen some statements recently from the leaders of the frontier labs: Jakub Pachocki at OpenAI had a statement to this effect, Dario Amade recently as well that was endorsed by some other people. Maybe you could explain to us what is the concern that people are raising about RSI, yeah, and just sort of walk us through the thinking on that.

Goldie: I mean, to be clear, people have been afraid of this for quite a while. Like I remember when we were at Google, there was an employee there, Blake Lemoine, who was very concerned that Lambda, which was the precursor to Gemini, was sentient, and he got Lambda its own lawyer and all of that. At the same time, obviously, there’s been a lot of progress since then.

Mirhoseini: The kind of concerns around AIs doing things they shouldn’t be doing, it’s real, and this is a real kind of reality—it’s a reality that we are all facing, and so it matters how we react to that, and how we kind of make sure that AIs are behaving the way we want as much as possible. Especially that there’s just so much positive impact to human life they can have to humanity.

Goldie: So we actually think that chips might be the main lever to kind of amplify, or diminish the harmful impacts of AI. So, for example, like if you help design chips for beneficial impact—like beneficial use cases, like in you know cancer research or climate change research, you could potentially amplify the positive benefit of AI, whereas if you withhold that technology from more harmful use cases…

Drew: I guess my thought is most chips right now are very general purpose.

Goldie: But they don’t have to be, and we think there’s a world coming where they’re not.

Drew: You can also imagine special purpose chips that are designed for harmful purposes, though, right?

Goldie: Yeah, and we don’t have to assist in that.

Drew: But like, how would you know? Like, if you have a chip that’s optimized for biology and protein folding, like you don’t know exactly whether it’s being used for new medicines and drugs or for bioweapons…

Goldie: Yeah, I mean, it’s definitely a challenge, but you don’t have to like help killer robot development or something like that. I think that’s easier in this domain than in other cases.

Mirhoseini: Yeah, and we can kind of, like we have a say in who we work with, who are our customers, and who we don’t sell our chips to. So there is that.

Drew: Yeah. I guess even for killer robots, they often seem dual-use to me.

Goldie: That’s true, that’s true.

Drew: It’s like if I have an inference chip that’s running on edge, maybe those drones are delivering needed supplies in faraway places. But yeah, to your point, you get to pick and choose your customers.

Goldie: Exactly.

Drew: Though ships do tend to circulate on the black market. We have seen a lot of that.

Goldie: Scary.

Drew: To your point about Blake Lemoine, I guess I tend to separate concerns about RSI from questions about consciousness and sentience—

Goldie: That’s fair.

Drew: Like I sort of treat them independently. Do you think that’s fair to do?

Goldie: I think it’s on a spectrum, maybe. Or I don’t know. Maybe maybe our consciousness is orthogonal to capability.

Drew: Or like very loosely correlated, or…

Goldie: Yeah.

Drew: Like in the space of possible minds, maybe they are correlated.

Goldie: I think there’s probably some kind of relationship there. Yeah, I think he was concerned that this model was already too much like a person or something like that, too intelligent. That’s what I got from it.

Drew: Yeah, I think so too. But I guess I mean when I’m thinking about the concerns that people raise with RSI, I think it tends to more have a sense of the pace of progress is just going too quickly. And if you take humans out of the loop and you have one AI system designing its successor, which goes on… like people sort of immediately have a sense that there’s something terminatory going on about this. And I see that as the source of the concern more than consciousness, though sometimes they do get conflated, I guess. Does that seem fair?

Mirhoseini: I think part of the alignment and intention of AIs, I think those are kind of could be related to these conversations around consciousness and whatnot. But at the same time, yeah, the role of humans as the models getting—self-improve and become better and better.

I feel like one thing that’s very clear is that there is a positive side of things that humans are becoming much more capable. They can do a lot of things. Maybe software, like app development, software engineering, these things are becoming—a lot of it is becoming more and more, like, commoditized. Like can be done by an AI, so that means they can use their creative side of things. They can think bigger. They can do things that they couldn’t do before. So there is that part of that. So there is like this enablement side of things, and maybe AI becomes a really good tool that kind of, like, realizes our imagination and gives life to things and build things that we always wanted to do, and now we have a tool to do that. So I think that part of like maybe there are a lot of things that are the execution part and the doing thing side of thing that AI can do and humans can still be on the creative side of things and together it could be…

Goldie: Also, you keep saying about like you know if we achieve—if recursive self-improvement starts happening, maybe this means consolidation by one company, but maybe in some ways, like we’re seeing increasing democratization of like of these capabilities, access to them, like small models that people can run locally, and like potentially more and more people could design their own hardware. Right now, there’s like a very small set of companies that are capable of building chips. We don’t think that has to be the case. So right, maybe in parallel, there’s also this proliferation or this democratization of AI and chips.

Drew: Yeah, I could see that happening as well. But I guess we’re imagining a world where it happens in parallel with the frontier systems rapidly improving.

Goldie: Both are happening right now.

Drew: They are. They are. Yeah. And potentially that gap between… Well, I guess I would maybe expect the gap between how long it takes for a new frontier model to come out versus, and then the period from that release to the first open source release that rivals its capability. I guess you could imagine that gap getting longer as the exponential goes on, because the pace

Goldie: It depends on the chips, basically, like how much compute they had. That’s like the biggest gap, at least for now.

Drew: It depends on the compute that went to training the…?

Goldie: Yeah, and I don’t think there’s, and there’s no open source model that was ever trained on anywhere close to as many chips as the closed-source frontier ones.

Drew: Yeah, yeah. You mentioned that there’s a question about alignment and like how alignment goes during this process of the recursive self-improvement. I guess some people have a concern that progress on capabilities, making models much smarter and more capable, could be comparatively easy compared to progress on alignment as this recursive self-improvement loop proceeds, and that’s one source of people’s concern here, I wonder—

Goldie: It’s just harder, right? It’s harder to define. Like, what does it mean? Like, as Azalia was alluding, like we have these objective measurements that we—and maybe it’s extremely complex to measure. But you can’t—Okay, what is the power consumption? What’s the area, like? It’s just a lot easier than like, is this model like a good entity, right?

Drew: Right, right.

Goldie: Like, how do you define that? I feel like constitutional AI was like a very thoughtful, maybe attempt or first step, but it’s just it’s very challenging.

Drew: Right. Compared to capabilities where you often have a clean number that you’re trying to make this number go down—

Goldie: Exactly.

Drew: Or you’re hill climbing this simple metric or some evals.

Goldie: It’s sort of like you know STEM. Like studying STEM to me seems way easier than studying humanities. There’s like a much clearer answer. Or maybe there are trade-offs, but it’s like it’s much clearer.

Drew: Yeah, definitely. Does that lead you to some view on how alignment is going to go, or how it should go to keep models aligned during that RSI process? I don’t know if you have a house view on this question.

Mirhoseini: I think right now as a company we really focus on this end-to-end co-design and chip design and chip design acceleration and automation. So that’s what I would say our focus is.

Alignment in general is one category of research and it’s super important, but maybe it’s unclear to all the entities how high of an ROI this research is compared to a capability research, so it might get less resourced or less like attention across the board on alignment and safety. This might be true. Like there’s a lot of research around it, but in terms of like resources, but in terms—just like we were saying, it could be this is like a much harder problem to solve, and there are scaling laws everywhere, so we need more compute allocated to do research and alignment.

And again, I think we are putting everything back to our chip design approach. Like having cheaper chips for any capability or any level of intelligence allows you to do that. At the end of the day, it’s all about how much focus we as people who work on AI put on a single problem, and every time that’s the case, a lot of progress is made, and that focus also comes with the compute and capital that they allocate there, and that all relates to the chips and all of that together.

So, in summary, there is like alignment, just like any other capability research, is like one direction, one capability in a way. It’s like alignment, whether the smaller is aligned or whether we can detect if it’s unaligned.

Drew: But fuzzier, and so maybe harder to automate—

Mirhoseini: It’s fuzzier.

Goldie: It’s much harder.

Mirhoseini: Maybe we need a lot of resources to solve this.

Drew: I think your point that the benefits of investing in alignment are more ambiguous: that sounds right to me. I’m thinking about some of the prosaic ways that models seem misaligned these days. Like even sycophancy, where the models would sort of flatter users, or sort of indulge or endorse users’ kind of delusional beliefs. That seems like it was going on for a long time, and then it got a lot of attention. OpenAI retired GPT-4o, which was especially guilty of this kind of behavior, and it seems to me that that problem has gotten a bit better since then.

Similarly, we see—I don’t know if you, you probably interact with coding agents a good amount.

Goldie: Yeah.

Drew: I think there’s similar, like small misalignments that we see there.

Goldie: Oh yeah.

Drew: Where sometimes they don’t follow instructions or they oversell their results, they get lazy, they pretend to have done things that they didn’t. And at most that’s kind of a nuisance, but then we see maybe similar behaviors with say like the Hugging Face hack, and now it’s like, “Oh, it’s a big deal. We should solve—”

And so there’s these maybe clarifying moments where the benefits of investing in alignment research become more obvious.

Goldie: That’s right, it’s like it directly translates into the ability to get the thing done that you wanted to get done. Yeah. I do think in terms of like that consciousness question, how it relates, like in terms of this spectrum of like to what extent these models are powerful tools that can be used by bad actors or used by good actors versus the extent to which they have their own intentions and those intentions are aligned or not. I feel like maybe consciousness relates to that.

Drew: Yeah, I could see that for sure. If people are interested to learn more about consciousness, we’re going to have an episode on this show about consciousness, so hopefully, we can do—

Goldie: Maybe we can learn too.

Drew: Yeah! I would be curious about your thoughts on it.

Maybe the last question to end on then is: after all this discussion about the pace of progress and what is already bottlenecking it quite a bit, and also our questions about alignment, what does that lead you to think about these proposals about pacing the frontier, which are sort of directly in response to concerns about the pace of RSI? It sounds like there’s a case to be made that maybe you just think RSI won’t go that quickly by default. But what do you think about these pacing the frontier proposals?

Goldie: I think it’s difficult in this—we’re like part of this larger system. I think you know Dario acknowledges it in his essay as well. Like the U.S. is not—is just one of many countries. These labs are just some of the labs that exist in the world. So I think that’s the biggest challenge. I think it’s like we do need to be very thoughtful here, but I think that’s the biggest…

Mirhoseini: Yeah, but there are things that are practical, like these embedded evals, and things like we would, as a lab, we would love to have the access to those and contribute to those. And these are very practical approaches to make the models…

Goldie: Exactly or like—

Mirhoseini: More and more aligned and…

Goldie: Working on restricting the black market for chip, things like that. That seems like a very concrete, useful, good idea.

Drew: Yeah, restricting the black market and then also maybe upgrading people’s security postures so the lab—the weights don’t just get stolen.

Golide: Exactly.

Drew: You can imagine—

Goldie: Makes a lot of sense.

Drew: Yeah, it could be bad if some bad actors get their hands on an AI system that’s very capable at chip design. Then maybe they kick off this kind of RSI loop themselves or something like that.

Goldie: Exactly, we take security very seriously for exactly that reason.

Drew: Well, thank you both so much for coming on the show. It’s such an interesting topic, and you have such an interesting and unique approach to it, so thank you for sharing your perspective and telling us about your work. Really appreciate it.

Golide: Thank you.

Mirhoseini: Thank you.

Drew: Thanks for listening to the information’s AI Deep Dive.