The Base Model Is Dead — Varun Singh, Arcee AI

Historically, base models were trained on vast amounts of web text like Common Crawl and WebText-2, aiming to capture a broad representation of human knowledge. This approach was dominant in early large language model development, with datasets like WebText comprising up to 85% of the training data for models such as GPT-3. The focus was on pre-training to accumulate world knowledge and build useful representations through next token prediction.

Recent advancements, particularly with OpenAI's 01 release and DeepSeek R1, have shifted the focus towards reinforcement learning. RL is no longer a mere 'cherry on top' for shaping model behavior but has become crucial for dramatically improving performance across various tasks. This shift enables models to interact with software environments, build applications through function calling, and learn complex interactions.

A significant change is the reduced reliance on web text as a primary training data source. While previously constituting up to 85% of the training mix, web text now represents only around 15% in newer models like MEI Thinking 1. This indicates a move away from solely relying on readily available web scrapes and suggests new approaches to data curation are emerging.

There's an ongoing debate about the role of synthetic data in training modern language models. The MEI Thinking 1 paper advocates for avoiding synthetic data and relying primarily on filtered web scripts to maintain a connection with human knowledge. However, this contrasts with other approaches, highlighting differing perspectives on how best to leverage diverse data sources to optimize model performance.

The speaker contemplates whether reinforcement learning (RL) might eventually reduce the reliance on supervised learning in language models. They note that human language presents a complex distribution, making it challenging to learn solely through RL. However, they suggest that we may observe a decrease in the importance of supervised learning as RL becomes more prevalent, highlighting the value of base models possessing fundamental skills for RL.

Some labs are experimenting with incorporating novel data into supervised learning processes. This 'novel' data refers to information the model hasn’t encountered before, pushing it beyond familiar patterns. An example given is reasoning traces, which often differ significantly from typical human-generated text output and can help models develop more advanced capabilities.

A strategy being explored involves training models to optimize performance during inference (test time). This is achieved by 'warming up' the model to specific compute schemes during supervised fine-tuning (SFT) or even pre-training. The goal is to improve efficiency and reduce latency when deploying these large language models in real-world applications.

The speaker argues that the role of base models is shifting from encapsulating general human knowledge and world priors to emphasizing reasoning and agentic behavior. While this simplification overlooks the complexity of reasoning agents, it reflects how chatbots are increasingly utilized – as interactive tools requiring problem-solving capabilities. This shift underscores the importance of designing base models with these specific skills in mind.