First Steps Toward Automated AI Research — Richard Socher, CEO Recursive AI

Richard Socher demonstrates Recursive AI's automated research system's ability to rapidly train a small chat model, achieving significant improvements in 'bits per byte' within a short timeframe. The system trained the model in less than 5 minutes and achieved a score of 0.91, surpassing the community’s previous best of 0.93 after extensive work. This showcases the potential for automated systems to accelerate AI development.

The system was used to optimize CUDA kernels, which are crucial for efficient GPU utilization in training and testing large AI models. After a few days of automated research, the system discovered kernels that outperformed those on Nvidia's benchmark website by a significant margin. This highlights the ability of RSI to identify improvements even in specialized areas where human expertise is typically required.

Richard Socher points out the surprising inefficiency of many Mixture of Experts (MoE) models, noting that even large clusters costing billions of dollars often have only 30% utilization. This highlights a significant area for improvement and suggests that automated research can play a crucial role in optimizing resource usage and reducing costs within AI infrastructure.

Socher argues that Recursive AI's automated research system (RSI) represents a potential 'S-curve' or exponential growth phase layered on top of existing advancements in AI. He emphasizes the vast remaining potential for improvement across various dimensions of intelligence, suggesting RSI could ultimately benefit not only AI but also science and technology as a whole.

Richard Socher introduces the 'Eureka machine,' a concept for an AI system designed to automate research and invent future technologies. He frames this as his personal equivalent of going to Mars, signifying a monumental undertaking. The goal is to mimic evolution's inventive capacity by automating scientific discovery and pushing the boundaries of human knowledge.

Socher emphasizes that evolution, a process spanning billions of years, has yielded remarkable advancements in biology and is now inspiring the development of AI. He suggests drawing parallels between evolutionary processes and AI design to create more effective systems, highlighting the potential for AI to mimic nature's iterative problem-solving approach.

Drawing from Marc Andreessen’s techno-optimist manifesto, Socher asserts that technology is the primary driver of economic growth. He counters concerns about AI displacing jobs by arguing that technological advancements will ultimately lead to massive economic expansion and widespread benefits for humanity, emphasizing a positive outlook on innovation's impact.

Socher illustrates the rapid pace of technological progress by contrasting advancements made between 1900 and 1969, highlighting humanity's transition from rudimentary flight to lunar travel. This example underscores how quickly technology evolves and suggests that current AI development may be on a similar trajectory, potentially leading to transformative changes within a single generation.