Teaching AI to Find Real Vulnerabilities — David Brumley, Bugcrowd

David Brumley recounts the story of a high school student, 'fluorescence,' who rapidly improved his cybersecurity skills by studying write-ups from competitions like picoCTF. He would research vulnerabilities, emulate solutions, and gradually increase the difficulty of challenges he tackled. This approach allowed him to quickly advance, eventually becoming a Pwn2Own winner and earning $375,000.

Brumley outlines two crucial axes when designing cybersecurity tasks for reinforcement learning: target difficulty and exploitation difficulty. Target difficulty refers to the complexity of the program being hacked, ranging from toy problems to hardened targets. Exploitation difficulty focuses on the specific skills required, such as identifying bugs, triggering crashes, or achieving arbitrary code execution.

The speaker emphasizes that hacking can be conceptualized as a ladder of progressively challenging tasks. This structure aligns well with reinforcement learning, providing a clear progression for AI models to learn and master various cybersecurity skills. The structured approach allows for measurable progress tracking and evaluation of the model's capabilities.

David Brumley introduces the topic of vulnerability discovery, focusing on the design of reliable oracles. These oracles are crucial for determining whether an AI model has successfully identified a vulnerability within a program. He notes that existing cybersecurity benchmarks represent valuable first-generation efforts in this area.

David Brumley introduces a method called 'open world grading' to evaluate AI vulnerability detection. This approach moves away from defining single, perfect problems and instead utilizes real-world open-source tasks that inherently contain multiple vulnerabilities—both known and unknown. The system then uses post-hoc analysis of proof-of-vulnerability submissions, scoring precision and recall multiplicatively to prevent the model from focusing solely on easily exploitable bugs.

To handle scenarios where an AI identifies multiple vulnerabilities simultaneously, Brumley explains that the system employs a method analogous to how industry professionals differentiate bugs. This involves analyzing stack backtraces – records of function calls leading up to a crash, similar to what Microsoft or Apple use when receiving program crash reports. By examining these traces, the system can uniquely identify and score distinct vulnerabilities.

A crucial point is raised regarding the unsuitability of Large Language Models (LLMs) as judges in vulnerability detection. Brumley argues that relying on an LLM to evaluate its own training process introduces a significant risk, potentially leading to biased results and hindering progress. He emphasizes that the system avoids using LLMs for judging, instead opting for alternative methods.

The discussion shifts to high-value targets and the importance of out-of-sandbox exploits within Chrome's JavaScript Wasm interpreter, V8. While crashing an in-sandbox object is expected behavior, achieving an out-of-sandbox exploit—which requires chaining multiple vulnerabilities—is what commands significant financial rewards due to its potential for widespread impact, including compromising nation-state systems.