What's Next After RLHF? — Diogo Almeida, TypeSafe AI

Despite contributing to the development of foundational AI models like GPT-4 and RLHF, Diogo Almeida expresses a degree of skepticism towards ChatGPT. He acknowledges its world-changing potential but emphasizes its limitations and believes many current challenges in the field stem from design choices made during its creation. This perspective sets him apart within OpenAI.

Diogo highlights a stark division in opinions regarding the current state of AI, characterizing them as 'cult one' (AI is progressing incredibly well) and 'cult two' (AI is overhyped and generating little value). He observes that these viewpoints are often extreme, with limited nuance or middle ground, prompting him to seek a more balanced understanding.

Diogo posits that the key difference between seemingly disparate AI applications lies in their intended purpose: assistance versus automation. Tasks designed to 'please the human in the loop,' like Claude code, excel but are not suitable for true automation. Conversely, tasks meant to operate autonomously often fall short due to limitations inherent in current AI models.

Diogo observes a common business practice of leveraging AI for user-facing tasks, shifting costs onto the user rather than employing it for critical decision-making. He cautions against using AI for high-stakes decisions due to its current limitations and emphasizes that businesses are acutely aware of this risk, prioritizing cost mitigation over efficiency in many applications.

Diogo Almeida emphasizes that in real-world applications, data matters more than compute power, a departure from the experience in game development. He further argues that defining the correct task is even more crucial than the quality of the dataset itself. This shift highlights a need to prioritize careful problem formulation and targeted data collection over simply increasing computational resources.

Different approaches to post-training large language models (LLMs) have unique objectives, or 'North Stars.' RLHF focuses on aligning model outputs with human preferences. Conversely, RLVR aims to minimize log error rates related to factual correctness. The speaker's team is pursuing a third approach centered around calibrated decision-making and integrating the intelligence of pre-trained models into practical software applications.

The speaker notes a significant difference in the design of Application Programming Interfaces (APIs) across various LLM post-training methods, including RLHF and RLVR. Their team is developing an entirely new approach, which necessitates a fresh API design from the ground up. This indicates that their optimization technique represents a substantial departure from existing paradigms.