Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase

The presentation uses an analogy of airport security to explain tiered checks. If a request's execution aligns with the established baseline, it bypasses more intensive checks (like tier two). However, if delays or deviations are detected, further investigation is triggered to pinpoint the source of the issue and understand the underlying cause.

To build robust anomaly detection systems, the team utilizes synthetic trace data. Over a period of seven days, millions of traces are injected into the system, including simulated anomalies. This allows them to train their models and refine thresholds before deploying changes to production environments, ensuring minimal disruption.

Ritvik clarifies the difference between anomalies and drift using a driving analogy. An anomaly is a single, unexpected event (e.g., traffic delay), while drift represents a gradual change in patterns over time (e.g., consistently longer commute times). Recognizing this distinction helps prioritize responses – immediate action for anomalies, and proactive adjustments for drift.

The speaker discusses a challenge where one system node delays event data, creating discrepancies. He poses the question of whether this delay should be considered a structural change, as it effectively reduces the number of nodes contributing data. Fine-tuning is needed to determine when such delays warrant classification as structural changes and avoid misinterpretations.

For use cases requiring real-time tracking, a tail-based system is recommended. This approach focuses on the start and end times of service requests for each node. The speaker emphasizes that when introducing new endpoints, establishing a fresh baseline is crucial to prevent overly generic anomaly detection.

Instead of applying uniform baselines for all post requests (e.g., across payment types), the speaker advocates for creating use-case specific baselines. This allows for more accurate anomaly detection tailored to different scenarios like payments, real-time transactions, or wire transfers. Such customization significantly improves accuracy and reduces false positives.

The speaker stresses the importance of data explainability, drawing an analogy to a doctor providing a health score without context. He argues that users need more than just numerical scores; they require detailed explanations and informative data to make informed decisions. This transparency builds trust and facilitates effective problem-solving.

Ritvik introduces the concept of execution graphs, which are short-lived representations of request processing flows. Unlike traditional graph databases used for querying relationships, execution graphs focus on identifying deviations from expected behavior within a single request's lifecycle. They simplify complex systems by representing them as Directed Acyclic Graphs (DAGs), enabling easier anomaly detection and automated troubleshooting.