Wearing the Agent: From Group Chats to Glasses — Sai Krishna Rallabandi

Sai Krishna observes that almost every agent built today caters to a single user, even in enterprise settings like programming assistants. This is a significant limitation as it doesn't address the complexities of group dynamics and shared information. The current focus on individual users creates challenges for future development and deployment.

Sai Krishna highlights that deploying agents in group settings introduces uniquely different challenges compared to single-user scenarios. These include considerations for privacy, information routing, and managing shared memory across multiple individuals with potentially conflicting needs or preferences. The shift necessitates a rethinking of agent design and security protocols.

Sai Krishna illustrates the potential of group agents through several examples, including a conference attendee request handled privately via direct message due to privacy concerns, calendar synchronization among family members, and an agent assisting a three-year-old in learning. These examples showcase how agents can facilitate communication, organization, and education within groups while respecting individual privacy.

Sai Krishna introduces 'Jetou,' a mythological reference representing a new security layer specifically designed for group agent memory. He emphasizes that securing an agentic system cannot rely on the same methods used to secure large language models, as agents actively operate on information and require more nuanced protection against hallucinations and misuse within a group context.

To manage bloated memory in group settings, a simple machine learning model or SLM can be used to identify important context and information. The paper 'Learning What To Forget' proposes using a relevance scorer that continuously evaluates the importance of data. This continual adaptation is crucial because context evolves over time in group work environments, requiring dynamic adjustments to what’s stored.

When serving stored memories, the choice between cloud-based and local serving impacts performance. Crucially, KV caches (Key-Value caches) – a common technique for accelerating model inference – can break down when dealing with complex memory manipulations. Therefore, developing an 'injection engine' that is aware of these KV cache limitations is essential to avoid instability.

Shared assistants create a new social contract regarding data privacy. Information shared within the agent can be either public or private depending on context, making it vital to consider the sensitivity of different types of information. For example, while a grocery list is benign, salary or health information requires careful handling and contextual awareness.

To address privacy concerns, the speaker suggests drawing inspiration from how the human brain works. Instead of storing all data directly, a common shared memory layer should be implemented with individual 'LoRA adapters' (Low-Rank Adaptation) trained for each user. This approach bakes in permissions at the model level using machine learning itself, rather than relying on code.