TheChek News Flow

China’s generative artificial intelligence startup sector is going through the “qualifying round,” according to Kai-Fu Lee, a venture capitalist and former president of Google China, who earlier this year founded 01.AI, a Beijing-based startup developing large language models, which last month raised a funding round at a $1 billion valuation.

The biggest internet companies in China, such as Alibaba, Tencent, Baidu and ByteDance, as well as dozens of startups, are all developing their own LLMs, in what local media have described as “a war of 100 models.” The Chinese government blocks OpenAI, Google and other U.S. internet services, so domestic firms are vying for AI supremacy. In an interview, Lee said companies are in the phase of trying to prove they have the technology to build a high-quality model. Those who pass that test will move on to the next phase, which will be about how to grow revenue and eventually generate profit.

“In China, we’ll eventually have a few big winners, a few decent exits, and most will end up either falling by the wayside or pivoting to something less ambitious,” like building applications and solutions for specific industries, Lee predicted.

His startup, founded in March, now has a team of more than 100 employees, all based in mainland China, mostly Beijing. Last month, 01.AI launched its first open-source LLM, Yi-34B, but the company won’t be relying on that model for future revenue. Instead, its business plan is to sell proprietary LLMs, mainly to customers in China. The startup is currently developing a new proprietary model with more than 100 billion parameters, according to Lee.

The startup faced some controversies last month after Yi-34B quickly climbed on top of the Hugging Face leaderboard for open-source LLMs. Developers’ inquiries revealed that Yi-34B had used Meta Platforms’ Llama open-source AI model without mentioning Llama. The drama ended as 01.AI renamed portions of Yi-34B to give credit to Llama and Lee apologized for the oversight. For more on that, see here.

Lee talked to The Information in a video call from his office in Beijing, where he discussed how 01.AI is coping with U.S. government chip export restrictions and global opportunities for Chinese AI firms. Here’s an edited version of the Q&A.

The Information: China currently has dozens of domestic competitors, if not more, developing LLMs. What will happen next?

Lee: I think China has seen this kind of situation many times before, with Groupon copycats, bicycle rental apps and, in the deep tech space, computer vision and speech recognition. When computer vision was proven to have made a breakthrough, everybody in China jumped in with every possible application, but most of them didn’t survive. China is a very, very competitive market, maybe even more so than the U.S.

The current competition in China is still the qualifying round. And the first test is: Which one of these 100 companies can make a high-quality model that creates real value? That means showing good performance because this is still an area where being better in technology makes all the difference when it comes to providing useful content and strategic insights. If you don’t have a good enough model, then it’s more of a toy than a technology that solves real problems.

Those who pass the technology test in the qualifying round will then move on to the next phase, which will be business value. What is your business model? How do you make money? Soon these companies will be evaluated on the basis of their profit and loss statements, and investors will be asking the same questions that they ask cloud providers, enterprise software companies and consumer apps. And if companies can’t answer those questions, their growth will come to an end. In the U.S., OpenAI has already shown that it has a world-leading technology and it can also generate revenue. OpenAI creates enough value so people are building apps on it and paying money for it.

In China, we’ll eventually have a few big winners, a few decent exits, but most will end up either falling by the wayside or pivoting to something less ambitious, like building applications and solutions for specific industries instead of trying to do the pre-trained large model, which will be increasingly expensive over time.

Chinese AI startups and their investors say China will develop its own ecosystem for generative AI models and applications, separately from the U.S. What do you think?

None of us wants a parallel universe. We all prefer to compete on a global scale and let the best company win. It’s more efficient that way. But we don’t control our fate in this case.

There are geopolitical issues in particular. If we wanted to enter the U.S. market, while there is no regulation that says we can’t, I don’t think we will get a lot of business because of the current—in my opinion, unfair—sentiment about Chinese software. So that’s just the practical reality we’ve come to accept.

We are open to business opportunities outside China, but we fully understand that some things are not in the cards. It’s not in the cards to sell our proprietary model to companies in the U.S. They are not going to buy it. We are not going to do anything silly.

China is obviously the big opportunity, but I wouldn’t write off other parts of the world as possible areas Chinese companies can reach. Silicon Valley’s approach is, generally speaking, a one-size-fits-all approach. It has more or less worked for companies like Facebook and Google, and [has] led to the American leadership. But this time it’s different because LLMs are trained on data. And data comes with questions about biases, ideologies and values. And American values aren’t preferred or even accepted in certain countries. China won’t be the only country. I would argue that the Middle East is another region that may want to think about religion and other issues differently. And I think it will lead to countries wanting more control over their models.

I do think there are opportunities to build special models that are different for different countries. This is something that Silicon Valley companies aren’t naturally going to be doing, because they feel that their values are the right values, and they want more people to try it and assimilate to it. And it’s a lot of engineering work to make different models for different markets. So Silicon Valley companies won’t be naturally inclined to build those models. And companies from other parts of the world, including China, may have opportunities to work on such models. Obviously they will have to win the trust of the users and the governments of the countries.

Chinese media have reported that your startup has managed to significantly lower the AI training cost for Yi-34B. How?

We have a very strong infrastructure team. It’s the biggest part of our workforce. One of the things I told my staff earlier is that every time you add a modeling person, you are spreading thin your [graphics processing units]. Every time you add an infrastructure person, you are getting more out of your GPUs. Of course you need a strong modeling team, but, from the get-go, we had a major priority to build a strong infrastructure team.

Infrastructure is the work of unsung heroes. They have to deal with hardware, software and huge amounts of data being transferred. They are simultaneously dealing with GPU, memory and networking, and any one of the three can become a bottleneck.

GPUs have a hard time scaling above several thousand. When you go from 2,000 to 8,000, you can’t just take the software and run with that, because the amount of networking demands change dramatically as you go to larger models and larger data sizes.

The infrastructure team, which has dozens of engineers, is the largest team now at 01.AI. For example, the kinds of work our infrastructure team has done include figuring out how to use FP8—a data format for Nvidia’s H100 chips—end to end, which leads to [a] substantial reduction in computation. The team figured out where to use FP8 and where to use other types and how to convert them in a seamless way. They also dealt with questions like what networking protocol we should use, how to optimize the compiler and how to deal with failing GPUs. GPUs actually fail pretty frequently. When a GPU fails, can you hot-plug? Unplug? We are still working on that. If your training stops for an hour because a [single] GPU fails out of a thousand-GPU cluster, to be able to hot-swap it will save you an hour a day. These little things will add up.

Another related thing is elastic training. That means if you have a cluster of 2,000 H100 chips, and you need 500 for a separate task, can you remove them between checkpoints and can you add them back later? These [tasks] are not what AI researchers have been trained to do. They are more related to networking engineers.

If the development of LLM is rocket science, the rocket will never fly without the engineers. SpaceX didn’t succeed just because of a bunch of researchers. It succeeded because it had a ton of very complex, intricate engineering.

The U.S. has restricted the exports of advanced semiconductor technology to China, including key Nvidia chips. How are you coping with this?

I’ve publicly said we have enough chips to last for 18 months. That’s basically the chips we acquired before the restrictions. We are definitely studying hard how to use Chinese chips. It’s not easy. It’s not fun. Programming them is not something we’re experienced at. But if that’s what we gotta do, that’s what we gotta do.

Nvidia has exceptionally good chips, but there could be an argument that a simpler chip could do the job at a much lower cost. But a major factor behind Nvidia’s strength is the whole ecosystem around its CUDA software libraries that makes programming easy. Engineers would basically revolt if you forced them to use non-Nvidia chips because they are so much less productive. But we are currently facing that predicament in 18 months, and obviously we have to start much earlier. If we can’t get access to Nvidia chips, we will look for simpler chips that are more focused on transformers, but they will be a pain to program. But if we have no choice, then we have no choice.

But Chinese engineers are known to be able and willing and do an excellent job in dealing with such engineering challenges that are considered laborious grunt work. It’s similar to what I said earlier about the work of our infrastructure team. Learning to program a new nonstandard GPU with very minimal libraries is also grunt work.

Chinese entrepreneurs are tenacious. Chinese engineers are hardworking. They are unafraid of grunt work. That’s the very reason why Meituan built a superior service, and why WeChat is a superior product.

Yes, these are difficult challenges, and you could say they are a waste of time and a lot of people’s energy. But these are the cards we are dealt, so we will do our best to play our hand.