Build for the Memo, Not the Demo — Shawn Chan, China Resources Holdings

Shawn Chan emphasizes the critical distinction between facts and guesses within AI-generated content. He explains that when AI systems seamlessly blend these two, committees responsible for approval are unable to discern their origin, leading to potentially flawed decisions. This blending can create a false sense of certainty, as exemplified by the phrase 'the company will likely receive approval next quarter' appearing as a fact.

A lawyer’s reliance on an AI chatbot resulted in the submission of legal arguments citing six non-existent court cases. This incident highlights a significant risk: AI can confidently fabricate information, complete with proper formatting and citations, making it difficult to detect falsehoods. The lawyer's attempt to verify the cases by asking the chatbot itself only confirmed the deception.

Chan argues that companies cannot absolve themselves of responsibility for their AI systems' actions. The airline chatbot incident, where a customer successfully sued the airline after receiving incorrect information, demonstrates this point. The airline’s defense – claiming the chatbot was a separate legal entity – failed and underscored the necessity of having a human accountable for AI-driven decisions.

To ensure reliability, Chan outlines five crucial fixes: each claim must have a traceable source with trust level; facts and guesses should be visually distinct; numbers need to automatically agree; contradictions should be flagged instead of smoothed over; and a human approval gate with logged changes is required. These measures prioritize transparency and accountability over superficial polish.

The speaker highlights a crucial distinction: confidence and accuracy are not synonymous. He observes that AI, like individuals in finance, can exhibit great self-assurance while being fundamentally incorrect. This is because AI quickly learns to mimic the appearance of certainty without necessarily possessing genuine understanding or factual correctness.

A key lesson learned from investment committee meetings is that challenging underlying assumptions is paramount. The speaker recounts an instance where a simple question about the origin of a number exposed flaws in a polished presentation, revealing more about financial realities than years of formal education ever could.

The speaker introduces the concepts of 'demos' and 'memos,' contrasting their purposes. Demos aim to create a fleeting impression, while memos must withstand rigorous scrutiny and defend against challenges. He uses the analogy of summarizing an email versus defending a mortgage application to illustrate this difference.

A significant market capitalization loss resulted from an incorrect statement in a promotional demo for a major tech company's AI assistant. This incident underscores the critical importance of verifying every claim, as even seemingly minor inaccuracies can have substantial financial consequences and highlights how demos fail to meet memo standards.