Data and Environment Curation for Post-Training LLMs — Mahesh Sathiamoorthy, Bespoke Labs

Bespoke Labs is an applied data research lab focused on helping enterprises and frontier labs access high-quality data and RL environments for post-training needs. They aim to bridge the gap between data creators and researchers, recognizing that understanding researcher needs is crucial for effective curation. This perspective informs their approach to creating datasets like Open Thoughts and contributing to projects such as Terminal Bench.

Mahesh explains a shift in how Large Language Models are evaluated. Initially, models were assessed based on their knowledge across various domains like STEM and humanities. However, the focus has now moved towards evaluating agents' ability to *do* tasks autonomously, leading to benchmarks that test agent capabilities rather than just factual recall.

The speaker highlights reliability as the primary obstacle preventing agents from operating autonomously for extended periods (hours, days, or weeks). He explains that agent failures often stem from errors like incorrect tool usage or flawed reasoning. Improving reliability can be achieved through techniques such as prompt engineering, harness updates, and crucially, post-training.

Despite advancements in compute power, model architectures, and training infrastructure, data remains the primary bottleneck for successful post-training of LLMs. This applies to both frontier labs with extensive resources and enterprises seeking custom models. Bespoke Labs' work is driven by this realization, focusing on research and development of high-quality datasets and RL environments.

Mahesh explains that adding tags to prompt-response pairs, specifically focusing on the form rather than specific numbers, significantly improved model performance. This technique led to notable boosts in compliance metrics, latency reduction, and increased throughput. The tagging approach allows models to be more easily updated as frontier models improve, reducing ongoing maintenance costs.

Bespoke Labs developed a data curation tool designed to streamline the process of fine-tuning models. This tooling allows users to leverage existing datasets like Hugging Face or integrate with collected logs, simplifying the creation of training data. The curator integrates with tools like Tinker and Fireworks, which were previously used for curating Open Thoughts.

Mahesh outlines a layered infrastructure approach critical for developing Reinforcement Learning models and post-training agents. This includes managing RL environments, measuring quality, tracking versions, providing sandboxes for rollouts (especially important for long horizon rollouts requiring checkpointing), and robust compute orchestration capabilities.

The speaker introduces 'Japa,' a technique utilizing Large Language Models to optimize prompts through reflection. This method allows for iterative improvements to system prompts and harnesses, demonstrating an innovative approach to prompt engineering that leverages the power of LLMs themselves.