Learning on the Job: The Future of Post-Training — Raymond Feng, Applied Compute

Raymond Feng explains that enterprises increasingly desire 'plug-and-play' agents, meaning they want to train custom models within their existing agent workflows. This necessitates post-training methods capable of adapting to diverse harnesses – the code and infrastructure surrounding an agent – even when source code access is limited. The ability to adapt to these external systems is a key challenge in modern AI development.

Applied Compute frames their post-training approach as analogous to human learning, emphasizing the importance of building foundational skills before tackling more complex tasks. This involves starting with simple, single-turn Q&A and gradually progressing to longer horizon synthetic environment tasks that require higher-order reasoning and tool usage. The framework prioritizes compounding understanding through iterative skill development.

The basic Q&A training setup involves an orchestrator that holds a task (prompt and answer), sends it to a model, receives the response, and then submits it for grading. A training engine uses these graded interactions to compute weight updates which are synced to inference engines, creating a continuous loop of improvement. The key is having graded chats in a specific format within a controlled environment.

To handle longer and more complex tasks, Applied Compute utilizes synthetic environments where the environment state resides outside of the training stack. This allows for multi-turn interactions involving tool calls and sandbox modifications. The resulting task traces are then graded and used to update model weights via GRPO (Generalized Reinforcement Proximal Policy Optimization), enabling learning in more dynamic scenarios.

Raymond envisions a future where models aren't limited to improving on specific tasks but instead function as agents capable of interacting and learning across diverse environments. This involves the model continuously evaluating its performance in different settings, essentially treating itself as an entity to be improved upon through every interaction it has. This shift moves beyond task-specific improvements towards a holistic self-optimization strategy.

A crucial element of this future is the model's ability to perform introspection – assessing its performance based on different interaction types. The system would automatically compute weight updates from these self-evaluations, leading to continuous improvement without explicit human intervention. This allows for a more dynamic and adaptive learning process compared to traditional methods.

Raymond highlights the current challenge of improving models as a “Whac-A-Mole” game. Focusing on individual tasks or failure modes requires constant data creation and model adjustments whenever new issues arise. A self-improving system, understanding all interactions within its environment, would circumvent this problem by proactively addressing emerging challenges.

Raymond concludes with a quote from a recent paper emphasizing the shift towards experience as the primary driver of AI improvement. He believes that experiential data will soon surpass the scale of human-labeled datasets currently used, marking a new era in AI development where continuous interaction and learning become paramount.