AI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j

The workshop is driven by the Graph Academy website, requiring enrollment with an email address. Participants need to set up their environment using a code spaces link provided on the website and obtain credentials including an Anthropic API key (or use a provided one) and a BigQuery access key. The setup process may take approximately five minutes.

To illustrate the concepts, the workshop uses a fictional company called Autofix Group, which is modeled after a national auto repair chain like Pep Boys. This scenario includes data on vehicle bays, maintenance manuals, safety bulletins, recalls, and logged repairs, with technicians (like Danny) performing work and leadership seeking to create an AI co-pilot for assistance.

While vector search and text-to-SQL tools can initially access data within a lakehouse, challenges arise when dealing with massive datasets (hundreds or thousands of tables, millions of documents). This can lead to inaccuracies and missed information, as the AI agent may struggle to provide the right context for complex queries.

Graph databases are valuable for addressing estate-level questions that go beyond simple queries, such as identifying missing documentation or uncovering common repair patterns. Unlike semantic search, which struggles with negative examples, graph traversal can reveal gaps and relationships across the entire dataset, enabling a more holistic understanding of the data.

The workshop emphasizes using agents to assist in building queries rather than manually coding them line by line. Neo4j CLY, a tool that allows agents to run queries directly against the database, will be utilized for this purpose. This approach leverages skills like Neo4j Cipher and Graph Data Science (GDS) to facilitate modern query construction and algorithm implementation.

The workshop workspace contains a 'skills' file pre-populated with code covering many common tasks. This includes files named 'outline,' 'search,' and 'theme,' representing shapes used to query BigQuery, as well as a database query file for interacting with BigQuery via simple SQL commands. These elements are designed to be customized with specifications for agentic coding.

The BigQuery schema is kept intentionally simple for the course, consisting of entities like work orders, vehicles, DTC codes, procedures, work order parts, and parts. These elements are linked through a primary foreign key pattern to facilitate querying and graph representation. A large bucket of PDF files represents unstructured data within the schema.

To enable agents to effectively join data, a graph representation of a semantic layer will be created using Neoarta, a Neo4j labs project. This approach aims to provide a structured and interconnected view of the data, facilitating more sophisticated querying and analysis by the agent.

The speaker guides users through setting up their environment, emphasizing the need to copy provided credentials. An environment file is crucial for storing API keys (Anthropic, BigQuery, Neo4j). If the environment file doesn't populate immediately, it’s likely due to background processes creating necessary files and scripts; troubleshooting might involve checking Anthropic's status.

The process of generating queries involves reading a metadata graph and creating SQL Python files. These files execute selects and joins to pull together tables, essentially answering a specific question. This demonstrates how Neo4j leverages existing data structures to automate query creation and simplify complex relationships.

Neo4j is strategically focusing on ontologies, semantic layers, and 'virtual graphs.' This approach aims to allow agents to understand how tables interrelate without needing to physically move data into a graph database. The goal is to provide a graph schema of the existing SQL database, enabling direct Cypher query execution for enhanced agent understanding and efficiency.

The speaker addresses why migrating large Online Transaction Processing (OLTP) datasets to a graph database isn't always straightforward. Terabytes of continuously updated data require complex synchronization mechanisms, making it more challenging than simply importing the data. This highlights the value of Neo4j’s virtual graph approach which allows querying existing databases without full migration.

The speaker introduces a method for navigating documents similar to how one would use a table of contents in a book. This approach utilizes an 'outline' or 'table of contents' structure, which includes hierarchical containment trees (folders and subfolders) and links between documents. Each element within this structure possesses a URI, allowing users to understand the context of a document based on its location within the hierarchy – for example, knowing that a TSB recall notice is part of the technical library and linked to a manual.

The speaker contrasts deterministic graph loading with more complex methods like using Large Language Models (LLMs) for entity extraction in graph construction. Deterministic loading involves creating a graph based on the inherent structure of documents, such as folders and sections, without relying on LLMs to identify entities. This approach is presented as faster and more efficient when dealing with documents that already possess significant internal organization and interlinking.

The graph structure consists of a containment tree reflecting the folder hierarchy within a document library. This tree organizes documents into sections, which can be nested at different levels. Additionally, 'named links' are used to provide ordering and connections between sections, similar to how Obsidian vaults utilize internal linking. The URIs associated with each node maintain hierarchical context, allowing for easy navigation.

Deterministic graph loading offers several advantages over LLM-driven approaches. It is more performant because it avoids the computational overhead of using an LLM, and it's often faster to implement. This method proves particularly beneficial when working with documents that already have a defined structure and interlinking, allowing for rapid setup and efficient traversal.

The speaker explains that the effectiveness of NLP techniques, particularly regex, depends heavily on the structure of the document sources. Well-structured documents like manuals allow for simpler approaches using regex and basic NLP. However, less structured or 'messier' data may necessitate more complex tools like gler or other loom technologies, representing stages of increasing complexity in data processing.

NeoCarta's primary responsibility is to create a semantic layer graph and provide an MCP (Metadata Connectivity Protocol) endpoint. This makes it model-agnostic, meaning it can function with various models as long as the MCP server is accessible. The skills framework and Neo Carta CLI offer alternative access methods beyond the standard approach.

When defining relationships in a graph data model, naming conventions are crucial. While there's an element of judgment involved, keeping names simple and consistent (like 'has') is recommended to avoid complexity, especially when dealing with hierarchical relationships like folder containment or document sections. Granular naming allows for more detailed queries but can complicate the overall data model and potentially hinder query generation.

The speaker clarifies that semantic search isn't intended to replace full-text search or other methods. Instead, it functions best in conjunction with them. Certain tasks, such as proving something doesn’t exist or understanding complex document relationships, are better handled through graph navigation and document traversal rather than relying solely on semantic search.

The speaker emphasizes the importance of accurate document naming for agents traversing knowledge graphs. Poorly named documents can lead to misinterpretation by the agent, hindering its ability to understand relationships between information. To mitigate this, search augmentation is recommended to allow the agent to examine document content directly.

A significant challenge in knowledge base management arises from outdated or drifted documents. The speaker suggests incorporating metadata like 'last updated' dates and designating authoritative documents to maintain data integrity. Proper link naming, similar to Obsidian’s synonym functionality, is also crucial for clarity.

Neo4j's graph loading process offers a deterministic approach, meaning the same documents will consistently produce the same graph structure. This allows for incremental updates; by modifying a single node and its associated links, you can update only that portion of the graph without reloading the entire dataset, provided dependencies are carefully managed.

The speaker introduces theme detection as a valuable application for graphs with extensive interlinking. Link community detection algorithms can reveal natural clusters and relationships within the data, surfacing previously unknown connections between concepts or documents. This is particularly useful in knowledge bases where topics naturally group together.

Zach explains that the `gamma` hyperparameter in the theme generation process controls the granularity of the resulting groups. Increasing `gamma` from its default value of 1 to 2, for example, resulted in splitting the data into 14 themes instead of the initial 13. This allows users to adjust the level of detail and specificity when analyzing their corpus.

The theme generation process doesn't assign names to themes; instead, it directly uses metadata from the documents and links. This means that the labels and identifiers within the generated themes are derived directly from the data itself, ensuring consistency and transparency in the analysis. This approach prioritizes stability and direct representation of the underlying information.

The Leiden algorithm's projection-based approach allows for creating a time series of theme IDs when data is constantly updated. This enables tracking how themes evolve over time, which is particularly useful in scenarios like fraud detection or anti-money laundering where patterns and clusters need to be monitored for changes. This temporal perspective provides valuable insights into evolving trends.

When dealing with very large graphs, it's essential to filter the view sent to the AI model to ensure manageability. Zach suggests setting cutoffs based on community size or conductance metrics to reduce complexity and focus on the most relevant themes. This optimization balances information richness with computational efficiency.

The speaker highlights the utility of graph databases in addressing queries that involve identifying gaps or mismatches within data. Unlike traditional lexical searches, graphs excel at uncovering patterns and anomalies where the desired information might be absent or unexpected. This is particularly valuable when investigating discrepancies between work order history and associated documentation.

As Neo4j's capabilities evolve, the system will increasingly prefer using custom Cypher queries written in Neo4j CLY. This shift indicates a move towards more sophisticated and tailored data exploration, allowing agents to leverage advanced querying techniques beyond basic prefix shapes for greater precision and flexibility.

The demonstration reveals that Neo4j's semantic expansion feature significantly enhances search capabilities. This technique allows the system to broaden its search beyond exact matches, uncovering related information and potentially identifying missing documentation or procedures. The hierarchical URIs enable more efficient traversal and discovery of linked data.

The speaker emphasizes that utilizing the outline (hierarchical structure) within Neo4j can improve search efficiency. When an agent leverages the outline, it can traverse links more effectively and avoid exhaustive full-text searches, leading to faster query resolution. This highlights a trade-off between comprehensiveness and speed.