Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI

Ari Morcos highlights that compute availability has become extremely scarce over the last six months, with H100 prices increasing by 40% from their lows at the end of the previous year. This scarcity is driven by increased model complexity and token usage, leading to potential limitations on access to frontier APIs.

The transcript states that reasoning models utilize eight times as many tokens compared to non-reasoning models, with a projected 5x increase in token usage within the next year. This exponential growth in token consumption exacerbates compute constraints and contributes to concerns about API access limitations.

Ari Morcos emphasizes that data quality serves as a 'compute multiplier,' enabling improved model performance with the same compute budget or achieving similar performance with reduced computational resources. He illustrates this concept using a schematic showing how better data quality can steepen the learning curve, represented by a shift from a gray to a blue curve.

Datology AI differentiates itself by refining existing data tokens rather than sourcing new ones. They operate as an 'oil refinery for data,' taking data from public, proprietary, and licensed sources and enhancing its quality through a process involving four key steps: cleaning, curating, creating, and composing.

The TR team applied a sophisticated post-training harness to a mid-train model, and the resulting gain nearly tripled compared to applying it to a default instruction tune model. This demonstrates that improving the initial policy of the model through techniques like post-training can dramatically enhance its inference capabilities even without changing the training data itself.

The RC team trained a large language model on 17 trillion tokens of publicly available data, achieving performance comparable to models like Open Frontier and GLM5. Remarkably, the total cost for training this competitive model, including salaries, compute, and R&D, was less than $20 million – demonstrating that high-quality data can significantly reduce training costs.

Ari emphasizes that focusing on 'signal per token'—i.e., data quality—is more valuable than simply increasing the number of tokens used for training. He states that repeating high-quality data is preferable to showing low-quality data, and ultimately concludes that data quality remains the single most underleveraged compute multiplier for model development.

Ari highlights that effectively leveraging data quality requires advanced techniques to score and understand data across multiple dimensions. This involves scaling these methods to massive datasets (pabytes) and represents an area of ongoing research, offering significant potential for improving model performance and efficiency.