From Signal to PR: Anatomy of a Self-Improving Agent — Jason Lopatecki, Arize

Jason Lopatecki explains that Arise’s initial attempt at building an agent was unsuccessful, describing it as 'frankly sucked.' This early experience, occurring approximately two years ago, proved invaluable in shaping their subsequent development. The company learned from this failure and incorporated those lessons into the design of Signal, their next-generation observability solution.

Observability is evolving from a human-centric model (clicking graphs in UIs) to an agent-driven approach, combining coding agents like those used with Pyroscope and Google Cloud. This shift leverages telemetry data – the 'smoke' emitted by systems – to enable AI agents to identify and implement fixes automatically. The speaker argues that this represents a significant change in the observability landscape.

While automation has improved, the primary bottleneck isn't necessarily applying fixes but rather ensuring their correctness. The speaker highlights the challenge of validating automated solutions – 'Do I have it right?' – and emphasizes that confidence remains a crucial factor in deploying agent-driven changes to systems.

Arise has inverted the traditional observability loop, where humans investigate and fix issues. Now, agents proactively identify potential problems and generate tickets or provide evidence for human review. This proactive approach allows humans to start with a deeper understanding of the issue, accelerating the troubleshooting process.

Jason explains that observability platforms are increasingly becoming integrated into the continuous loop of fixing issues, rather than just providing signals. This integration aims to connect local debugging experiences and evaluations directly to automated fixes or at least provide a starting point for developers. The goal is to streamline the process from identifying problems to implementing solutions.

Responding to a question about using cloud code directly, Jason emphasizes that effective automated fixes require well-designed skills. These skills are crucial for data acquisition—specifically, identifying the right traces and organizing them into files suitable for processing. He highlights that even with powerful models like Claude, successful automation depends on carefully crafted skill surfaces.

The speaker clarifies the role of 'evals' in the system, describing them as data layered onto production traces – what they call ‘online evals.’ These evaluations provide additional information to the agent beyond raw trace data. They are essentially AI-powered assessments that run periodically and assess a system’s performance, allowing for proactive identification and mitigation of issues.

The speaker notes the scalability of 'Element as a Judge,' an AI layer used for evaluations. They mention that some customers apply this technology across their entire dataset, enabling continuous assessment and proactive identification of potential issues. This demonstrates the ability to run assessments at scale while providing valuable insights into system behavior.