Scaling to Long Horizons — Ross Taylor & Chengxi Taylor, General Reasoning

When dealing with long horizon tasks, inference times can extend to weeks or even longer. This significantly exceeds the typical eight-step policy constraint used for off-policy learning. Consequently, GPUs are forced to remain idle while waiting for these lengthy inferences to complete, representing a major inefficiency.

To mitigate the GPU idling issue during long horizon inference, a value model can be employed for bootstrapping. This involves generating expectations before the episode's end, acting like dopamine in the human brain to facilitate model training. However, this approach introduces a trade-off by potentially introducing bias from the value model itself.

General Reasoning has developed Open Review, a platform hosting over 350 environments accessible through a single API endpoint. This infrastructure is crucial for long horizon tasks and is currently utilized internally by General Reasoning as well as several frontier and new labs for reinforcement learning research.

The speakers emphasize that tackling long horizon challenges is not merely an engineering problem but requires a fundamental shift in mindset. To address humanity's most significant problems, embracing this long-horizon perspective and adapting algorithms, environments, and computational approaches becomes essential for progress.

Ross highlights that small, focused teams with aligned goals can achieve remarkable results, even within the context of scaling AI development. He points to his experience at Meta where a relatively small team was responsible for significant advancements in open-weight models like Llama 2 and Llama 3. This suggests that concentrated effort and shared vision are vital ingredients for innovation.

Ross emphasizes that Reinforcement Learning from Human Feedback (RLHF) was the crucial element transforming Large Language Models (LLMs) into usable products. He contrasts Galactica, a model lacking RLHF, with ChatGPT, highlighting how the feedback pipeline enabled LLMs to cross what he calls the 'Rubicon' and become widely adopted by billions of users. This demonstrates that even advanced base models are insufficient without RLHF.

Despite the negative public reaction and association with Meta, Galactica demonstrated superior performance compared to models like Palm and GPT-3.5 in scientific domains. It achieved 68% accuracy on research paper tasks versus GPT-3.5's 49%, and outperformed Palm (540 billion parameters) on chain-of-thought reasoning tasks. This underscores the potential of well-designed base models, even when overshadowed by other factors.

Ross notes the paradox that Galactica was a state-of-the-art model despite its public downfall. It outperformed larger models like Palm and Chinchilla with significantly less compute in scientific domains, demonstrating the power of base models. However, this also reinforces his earlier point that even exceptional base models require RLHF to reach their full potential.