Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute

Alex Shaw references a tweet from 2018 describing a common software engineering experience: receiving a large pull request and reviewing it, finding beauty in the code but noting a lack of consideration for extensibility. This evokes a sense of nostalgia, highlighting how quickly practices evolve even within relatively short timeframes – only six to eighteen months had passed since this described scenario was commonplace.

Shaw contrasts a 2018 program using regular expressions (regex) for phone number extraction with a 2026 version utilizing a large language model (LLM). While the LLM-powered code is potentially more powerful and adaptable, it introduces uncertainty; running it multiple times may not yield identical results due to the inherent stochasticity of these models.

Building on Francois Chalet's original tweet, Alex Shaw expands the concept that agentic coding is a form of machine learning. He argues that agent performance itself should be treated as a 'black box artifact', meaning its behavior and generalization are best managed through empirical evaluation – similar to how we approach evaluating ML models.

Alex Shaw outlines a direct comparison between machine learning workflows and agent building processes. Training data becomes environments, test/validation sets become evaluations (environments), model weights transform into skills, prompts & tools, loss functions equate to environment rewards, backpropagation is analogous to context-based optimization or pull requests, and overfitting manifests as reward hacking.

A significant emergent use case for Harbor is what the speakers call 'agentic map reduce,' which involves running a large number of agents on distributed compute sandboxes in parallel and aggregating the results. This process leverages Harbor Exec, a feature built specifically to address this need, allowing users to process numerous agent runs and extract insights like recurring mistakes or failure categories. The process utilizes tools such as Cursor CLI for fast processing and Fable 5 for accurate summarization.

Several organizations are utilizing Harbor to enhance their model training processes. Poolside uses Harbor for all of its model evaluations, while others leverage it for reinforcement learning by extracting reward signals from agent trajectories. This demonstrates Harbor's versatility beyond simple evaluation and its integration into the full lifecycle of AI model development.

The presentation highlights a rapidly growing ecosystem built around Harbor, with numerous benchmarks and tools integrating with it. Examples include Frontier Suite for software engineering, Banker for investment banking, Rune Bench for Runescape gameplay, and Scale's Sweet Atlas suite. This widespread adoption indicates Harbor’s increasing importance in the agent-based AI landscape.

The speaker emphasizes that the Harbor team is actively seeking new members, indicating a period of rapid growth and development for the platform. To facilitate this recruitment effort, they direct interested individuals to their Twitter account, highlighting the community's role in shaping Harbor’s future.