Ex-DeepMind and Meta AI Researcher Raises Nearly $50 Million To Help Developers With LLMs’ Secret Sauce
For all that’s leaked about the sizes and architectures of the advanced large language models developed by OpenAI and its ilk, we still don’t know what data they were trained on.
A pessimist might say that’s because of the numerous lawsuits LLM developers would face if the amount of copyrighted content they were using was revealed, but a more generous explanation would be that it’s because the makeup of their training data is these companies’ secret sauce.
The importance of data explains why Ari Morcos, a former DeepMind and Meta Platforms AI researcher, co-founded DatologyAI in 2023 alongside ex-Twitter engineering lead Bogdan Gaza and former MosaicML head of data research Matthew Leavitt. The three aim to help researchers and developers better curate their training datasets to improve the quality of their models. Investors are responding. Just three months after announcing its $11.65 million seed round, the Redwood City-based startup has raised an additional $46 million in Series A funding led by Felicis, with participation from Radical Ventures, Amplify Partners, Elad Gil, M12 and Alexa Fund.
DatologyAI, which currently has 11 employees, aims to reduce the amount of human decision-making—which can often be biased or time-consuming—needed in data curation, Morcos told me. That’s a different strategy from the more hands-on approaches of other data curation startups, where employees manually look at their customers’ private data to see where gaps exist. That’s made them more like consultancies than software companies in many cases, VCs tell me.
Instead, DatologyAI uses algorithms to automatically determine how much data a model needs to understand a certain concept. For example, a model will need more examples of a complex concept (like dogs, which vary in appearance) than a simple one (like elephants, which look relatively similar to each other) to understand them, Morcos said. These algorithms also ensure that models see enough rarer “edge cases” and that the data is divided into more manageable chunks during the training process.
Equally important to the training data itself is the order that it’s presented to the model in. Typically, researchers will show LLMs simpler concepts and lower-quality data first, when the models “know” less and can glean more information from the weaker data, Morcos said. Later on in the training, they will show data that’s more relevant to the models’ specific purpose, like lines of software code for a code-generation model, he said.
Though not uncommon for the hot AI market, I did ask Morcos about the reason behind his startup’s back-to-back rounds. DatologyAI is in an interesting position, he explained: Though it doesn’t have the computing needs of a foundation model developer like OpenAI, it does actually need to train models to confirm that the changes it makes to training data lead to better performance. That means that it needs more computing power than other startups developing tools for developers—and therefore, more capital. (Though I’m sure most AI startups wouldn’t say no to more money regardless.)
Morcos also pointed out that, as we’ve been reporting for months, AI talent can be incredibly expensive, especially when a company is competing with the likes of Google, OpenAI and Meta. So some of that capital will also be put towards attracting AI researchers and engineers, he said.