Key Moments
How to Build the Future: Demis Hassabis
Y CombinatorKey Moments
AGI may arrive by 2030, forcing deep tech journeys to account for its presence, but key challenges like continual learning and long-term reasoning remain unsolved.
Key Insights
The current AI paradigm, including large-scale pre-training, RLHF, and chain of thought, is likely to be part of AGI architecture, but continual learning, long-term reasoning, and memory are still unsolved crucial components.
The brain's ability to integrate new knowledge gracefully, as seen in sleep consolidation and episodic memory replay, inspires AI's 'dream cycles' and experience replay, but current methods like stuffing into context windows are still 'duct tape' solutions.
Reinforcement learning and search, pioneered in AlphaGo and AlphaZero, remain highly relevant and are being re-examined at scale for today's foundation models, suggesting RL may still be underrated.
DeepMind's strength lies in not only building frontier models but also rapidly distilling their power into smaller, hyper-efficient 'flashlight' models (like Gemma) that offer significant cost and speed advantages, potentially reaching 95% of frontier capabilities at a fraction of the price.
While AI can amplify engineer productivity thousands of times, there's a missing 'craft and human soul' element in creative applications like game development, indicating that AI tools are still maturing for complex creative tasks.
AlphaFold-style breakthroughs are likely in scientific domains characterized by massive combinatorial search spaces with clear objective functions and sufficient data or simulators, such as drug discovery and potentially virtual cell modeling within 10 years.
Unsolved frontiers in the path to AGI
Demis Hassabis believes that while current AI paradigms like large-scale pre-training, RLHF, and chain-of-thought reasoning will form the backbone of Artificial General Intelligence (AGI), several critical components remain elusive. These include continual learning, long-term reasoning, and robust memory systems. Hassabis estimates his own AGI timeline around 2030 and emphasizes that any deep tech journey initiated today must factor in the potential arrival of AGI mid-development. He posits that while existing techniques might scale with incremental innovation, there's a possibility of one or two significant new ideas being required. The challenge extends to ensuring systems are consistent and can actively solve problems, with agents being a promising avenue, though still in their nascent stages.
Biologically inspired memory and learning
The concept of 'continual learning' in AI draws parallels to biological memory consolidation. Hassabis's PhD work focused on the hippocampus and how the brain integrates new information gracefully. This biological insight inspired early AI techniques like 'experience replay' in DeepMind's first Atari program (DQN) in 2013, where successful game trajectories were replayed. While current methods often rely on stuffing vast amounts of data into large context windows, which Hassabis likens to 'duct tape,' he acknowledges its limitations. Even with potentially massive context windows, the cost of retrieving relevant information remains a non-trivial challenge, indicating significant room for innovation in AI memory systems.
The enduring relevance of reinforcement learning and search
DeepMind's foundational work in reinforcement learning (RL) and search, exemplified by AlphaGo and AlphaZero, continues to be central to their current development, particularly in building Gemini. Hassabis describes these systems as 'agents' capable of autonomous goal achievement, planning, and decision-making. While initially developed for games, these principles are now being generalized to world models and language models. He notes that much of the progress in modern large language models, including chain-of-thought reasoning, can be traced back to the pioneering work done with AlphaGo. DeepMind is actively revisiting these earlier ideas, scaling them up and augmenting them with techniques like Monte Carlo search, suggesting RL is far from being an outdated approach and will be key to future advancements.
The power and efficiency of distilled models
A core strength of DeepMind and Google is their ability to distill the capabilities of massive frontier models into smaller, highly efficient 'flashlight' models. These smaller models, like Gemma, achieve a significant percentage (reportedly 95%) of the performance of their larger counterparts but at a fraction of the cost and latency. This is crucial for serving billions of users across Google's products. Hassabis sees no theoretical limit yet to how much intelligence can be packed into smaller models, with the implication that engineers can operate at vastly increased productivity levels. These efficient models are also vital for on-device or 'edge' AI applications, enhancing privacy and security, particularly in areas like robotics and personal assistants.
Limitations in reasoning and agent capabilities
Despite impressive 'chain-of-thought' reasoning, current AI models still exhibit 'jagged intelligence,' making basic errors that a human might not. Hassabis points to limitations in introspection and consistency, giving the example of AI struggling with games like chess where it might identify a blunder but has no better alternative and proceeds with it anyway. He believes agents, while foundational for AGI, are currently held back by the lack of continual learning and adaptability to context. While many are experimenting with agents, the output justifying the extensive computation is not yet consistently visible. Hassabis suggests that truly revolutionary creative output, such as a hit game developed entirely by AI, is still missing, hinting at a need for more than just brute-force computation.
The promise and challenges of open source and multimodality
DeepMind is a strong proponent of open source and open science, exemplified by the free release of AlphaFold and the accessible Gemma models. The rapid download rate of Gemma (40 million in two and a half weeks) highlights the demand for capable, locally runnable models. Hassabis views this as important for establishing Western open-source stacks. He also emphasizes Gemini's multimodal design from inception as a quiet advantage, enabling better world models, robotics applications, and the understanding of physical contexts crucial for real-world digital assistants. This multimodal capability is seen as a competitive edge for understanding intuitive physics and dynamic environments.
The future of scientific discovery with AI
Hassabis views AI as the ultimate tool for scientific advancement, aiming to solve 'root node problems' that unlock new avenues of discovery. AlphaFold, used by millions of researchers, is a prime example, with suggestions that nearly every future drug discovery will involve it. He anticipates similar breakthroughs across material science, climate modeling, mathematics, and drug discovery within the next few years. The key to AlphaFold-style success lies in identifying problems with massive combinatorial search spaces, a clear objective function, and sufficient data or simulators. He believes AI is approaching the capability for genuine scientific reasoning, potentially leading to hypothesis generation and even the creation of new scientific challenges, although the 'Einstein test' of replicating a genius's breakthrough moment remains a benchmark.
Advice for building at the AI frontier
For aspiring builders, Hassabis advises focusing on deep tech problems that intersect with AI, especially in areas involving 'the world of atoms' like materials or medicine. These interdisciplinary fields offer more defensibility against rapid AI model updates. He stresses that worthwhile endeavors are rarely easy and require genuine passion. For those embarking on a deep tech journey, factoring in the potential arrival of AGI around 2030 is critical, suggesting that specialized AI tools might be utilized by general AGI systems rather than being integrated into one monolithic brain. He also encourages imagining a future where AI is the norm and building useful systems that could be leveraged midway through that evolution.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
Building the Future with AI: Key Principles
Practical takeaways from this episode
Do This
Avoid This
Common Questions
Demis Hassabis identifies continual learning, long-term reasoning, and certain aspects of memory as ongoing challenges for AGI. He believes these are crucial for systems to become more consistent and capable of deeper problem-solving.
Topics
Mentioned in this video
An AI research lab, now part of Google, dedicated to solving intelligence and its applications in science.
A renowned startup accelerator that hosts 'YC Startup School' and provides mentorship to early-stage companies.
A spin-off from DeepMind focused on drug discovery using AI, building upon the success of AlphaFold.
A game-playing AI developed by DeepMind that famously beat a world champion at Go, pioneering agent systems.
A groundbreaking AI system from DeepMind that solved protein structure prediction, made freely available to scientists.
Google's advanced multimodal AI model series, built with the goal of achieving artificial general intelligence.
Deep Q-Network, an early Atari-playing AI program developed by DeepMind that used experience replay.
A family of open and accessible AI models from Google, designed to be efficient and runnable on local devices.
A world model built on top of Gemini, likely used for robotics and understanding environments.
A StarCraft-playing AI developed by DeepMind, building on techniques used in AlphaGo.
The second iteration of AlphaFold, which achieved significant breakthroughs in protein structure prediction.
A large language model developed by Anthropic, mentioned alongside Gemini as a general-purpose AI system.
Rapid Eye Movement sleep, a stage of sleep associated with vivid dreaming and memory consolidation.
A figure from Greek mythology who stole fire from the gods for humanity, used as a metaphor for powerful, potentially dangerous technology like AI.
A major unsolved problem in computer science and mathematics concerning the relationship between computational complexity classes.
Nuclear fusion, a potential source of clean energy that could significantly reduce energy costs.
More from Y Combinator
View all 584 summaries
40 minReplit's CEO On The Only Two Jobs Left In The Company Of The Future
22 minHow to Make Claude Code Your AI Engineering Team
44 minHow Stripe Built Their New Website
50 minRobots Are Finally Starting to Work
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free