What are the most impressive things robots can do today?

Boston Dynamics robots, particularly in their parkour-like movements and stair-climbing abilities, are highly impressive. Interacting with robots like SpotMini or Pepper can also evoke a psychological connection, even when their behavior is scripted or limited.

Can reinforcement learning systems learn to be more interactive and 'person-like'?

Yes, if the objective function is formalized to include human-like interaction. Reinforcement learning can optimize for a robot being 'fun to be around' and naturally acquire interactive features, potentially learning from comparative feedback ('I like this better than that') rather than explicit numerical scores.

Why is reinforcement learning so sample-inefficient, especially with sparse rewards?

Sparse rewards mean a system might take many actions before receiving any feedback. This makes it hard to attribute the reward to specific actions. RL needs numerous experiences to consistently correlate actions with outcomes, despite this inefficiency often being less pronounced in practice than theory suggests.

What are the main challenges for AI in handling real-world problems with long time scales?

The real world involves extremely long time scales, from muscle contractions to life decisions like pursuing a PhD. Credit assignment becomes incredibly difficult across these vast temporal gaps, requiring hierarchical reasoning far beyond current RL capabilities.

How important is transfer learning for creating general AI systems?

Transfer learning is crucial. Early successes like AlexNet showed that models trained on one task could be fine-tuned for others. Scaling up models and training them on vast datasets, as seen with language models, further enhances their ability to generalize and transfer knowledge to new tasks.

What are the most promising research directions for robot learning: imitation vs. self-play?

Self-play is promising because it inherently provides a learning signal from both success and failure against oneself, leading to faster learning. Imitation learning, especially through 'third-person watching' (where robots learn from human demonstrations), also offers high-quality data and rapid skill acquisition.

How can we ensure AI safety, especially for robots operating in the physical world?

Safety concerns include physical damage from strong robots or autonomous vehicles. While simulation is key for testing, developing proper 'unit tests' or representative tests, similar to how humans are evaluated for driving licenses, is an ongoing research challenge to ensure updated software doesn't introduce new risks.

Can AI systems be taught 'kindness', or even 'love'?

While AI systems don't experience emotions like humans, they can be programmed to optimize for objectives that elicit positive human responses. Just as dogs provide affection, AI could potentially achieve similar levels of perceived affection and contribute to human happiness, though the ethical implications are significant.

Key Moments

Pieter Abbeel: Deep Reinforcement Learning | Lex Fridman Podcast #10

Lex Fridman

Science & Technology6 min read43 min video

Dec 16, 2018|80,537 views|1,526|52

Save to Pod

Key Moments

On this page

TL;DR

Pieter Abbeel discusses deep reinforcement learning, robot interaction with humans, and AI's future.

Key Insights

Robots beating top athletes like Roger Federer face significant hardware and software challenges, with hardware potentially being the primary bottleneck.

Human-like psychological interaction with robots is an emergent property that AI, particularly through reinforcement learning, can optimize for if human feedback is formalized as an objective.

Reinforcement learning is effective despite sparse rewards because it can isolate consistent patterns associated with higher or lower rewards over many experiences.

Achieving true generalization, especially in complex, real-world scenarios with long time scales, requires hierarchical reasoning capabilities beyond current RL algorithms.

Transfer learning is a major success, with pre-trained models like AlexNet and large language models demonstrating reusable knowledge across tasks.

Meta-learning approaches, like optimizing for desired outcomes without explicit hierarchy design, show promise for developing emergent hierarchical concepts.

Self-play offers a highly efficient learning paradigm for RL problems by providing natural opponents and clear signals from success and failure.

Imitation learning, especially through third-person demonstrations translated by meta-learning, allows robots to learn new skills quickly.

Simulation is crucial for AI safety and testing, but an ensemble of diverse simulators might be more effective than a single, highly precise one.

AI safety involves both unintended physical harm and the broader long-term concerns of superintelligence, with current testing methodologies for robots still underdeveloped.

The evolution of human behavior suggests a long-term trend towards increased cooperation, which could potentially be mirrored in AI's development with kindness as an objective.

ROBOTICS CAPABILITIES AND CHALLENGES

The conversation opens with a speculative question about robots playing tennis at a professional level, highlighting that while AI (software) plays a role, significant hardware advancements in locomotion and manipulation are essential. Boston Dynamics' robots are noted for impressive physical feats like parkour, though the extent of true learning in their current capabilities is uncertain. The complexity of tasks like swinging a tennis racket to achieve human-level precision and spin is acknowledged, suggesting it's achievable through reinforcement learning with extensive trial-and-error, possibly augmented by simulation.

THE PSYCHOLOGY OF HUMAN-ROBOT INTERACTION

A fascinating aspect explored is the human tendency to anthropomorphize robots. Even with limited or scripted interactions, people often develop a psychological connection, attributing personhood and emotions to machines. This emergent psychological engagement presents an opportunity for AI research. Pieter Abbeel suggests that reinforcement learning systems could be optimized to foster such positive human interaction, potentially by formalizing human feedback as a reward signal, leading to robots that are more interactive and perhaps even 'likable'.

THE MECHANICS AND EFFICIENCY OF REINFORCEMENT LEARNING

The discussion delves into why reinforcement learning (RL) works, especially with sparse rewards. The core idea is that RL systems, over numerous trials, learn to associate sequences of actions with rewards. Even if a reward is delayed, comparing successful and unsuccessful sequences helps the algorithm adjust its policy. Abbeel explains that deep neural networks, by providing piecewise linear feedback control and effectively tiling the state space with shared expertise, contribute to RL's surprisingly efficient learning process, leveraging the strengths of linear control in complex systems.

NAVIGATING REAL-WORLD COMPLEXITY AND HIERARCHICAL REASONING

The real world presents challenges far exceeding current RL capabilities, particularly in terms of time scales and credit assignment. Abstract human decisions like pursuing a PhD are vastly removed from the low-level muscle contractions that execute actions. Addressing this requires advanced hierarchical reasoning, which is not yet fully developed. Traditional AI systems had reasoning but lacked real-world grounding; deep learning provides perception, creating an imperative to integrate these aspects. Approaches include bolting deep learning onto traditional systems or exploring information-theoretic methods for high-level action choices.

META-LEARNING AND THE QUEST FOR GENERALIZATION

Instead of explicitly designing hierarchies, meta-learning offers a path to optimize for goals like faster learning, which is what hierarchies aim to achieve. The RL^2 (learning to RL) approach, where the system learns how to learn, has shown emergent hierarchical-like behaviors, such as consistent navigation in mazes. This aligns with the broader challenge of transfer learning and generalization. Significant progress has been seen since AlexNet, where pre-trained models can be fine-tuned for new tasks, indicating that learned representations are reusable and transferable.

THE SPECTRUM OF GENERALIZATION AND THE SEARCH FOR FUNDAMENTAL PRINCIPLES

While deep learning excels at pattern recognition and task-specific mastery (e.g., predicting current solar system motion), achieving deeper generalization—like predicting the effects of entirely new physical phenomena (e.g., a new planet)—remains a frontier. This requires understanding fundamental, simpler explanations, akin to discovering the 'E=mc^2' for learning. The brain's modularity offers a potential principle: designing AI systems with similar modularity could foster more robust and adaptable capabilities, enabling them to generalize beyond mere pattern matching.

SIMULATION, IMITATION LEARNING, AND SELF-PLAY

Both imitation learning (learning from demonstrations) and self-play are promising research directions. Self-play is highly efficient because it provides continuous learning signals by pitting agents against themselves, inherently demonstrating both success and failure. However, many problems aren't easily framed as self-play. Imitation learning, particularly third-person demonstration where a robot learns to translate human actions into its own movements (akin to machine translation), allows for very rapid skill acquisition. This approach, especially when combined with meta-learning, is a major breakthrough for teaching robots complex tasks.

THE ROLE OF SIMULATION IN AI DEVELOPMENT AND SAFETY

Simulation plays a critical role in both training and testing AI, particularly for safety. While perfecting a single, highly precise simulator is challenging, an ensemble of diverse simulators might offer a more robust approach. By training across multiple representative simulators, AI systems could learn behaviors that generalize better to the real world, which itself can be viewed as one instance within a distribution of possible environments. This ensemble method helps mitigate the limitations of any single imperfect simulator.

ADDRESSING AI SAFETY AND TESTING METHODOLOGIES

AI safety concerns range from immediate physical harm by physically capable robots to the long-term existential risks of superintelligence. Current testing methodologies for AI, especially for autonomous systems like self-driving cars, are still underdeveloped compared to human licensing. While humans undergo limited tests, we have an implicit trust in their general capabilities. For AI, developing rigorous 'unit tests' and robust validation methods to ensure updates improve performance without introducing regressions is a significant research challenge.

THE POTENTIAL FOR KINDNESS AND LOVE IN AI

The conversation touches on whether AI can be taught kindness and even love. Human evolution has predisposed us to certain social behaviors and innate drives like pain and hunger, which facilitate learning and social cohesion within tribes, though inter-group kindness requires explicit learning. Steven Pinker's work suggests a historical trend towards decreased violence. Abbeel speculates that AI could achieve high levels of affection, comparable to human-animal bonds, by optimizing for objectives related to positive human interaction, raising questions about the desirability of such deep AI-human connections.

Mentioned in This Episode

●Products

●Software & Apps

●Companies

●Books

●Concepts

●People Referenced

Common Questions

Achieving this requires significant advancements in both hardware (for agility and movement) and software (for racket swinging precision and spin). While hardware might see progress in 10-15 years, a full humanoid robot player is likely further out, though non-bipedal robot solutions on wheels could be developed sooner.

Topics

Ai Safety AI & Machine Learning Technology & Innovation Science & Mathematics Deep Learning Human-robot Interaction Imitation Learning Transfer Learning

Mentioned in this video

Software & Apps

AlexNet

An early deep learning model from 2012 that represented a breakthrough in image recognition and demonstrated the power of transfer learning through fine-tuning for new tasks.

Pepper

A humanoid robot that was scripted to have a child-like personality and was observed to be very interactive, making it hard not to perceive it as a person.

People

Paul Christiana

Collaborator at OpenAI who worked on projects like teaching a one-legged robot (Hopper) to do backflips using comparative feedback.

Vladimir Vapnik

A statistician in statistical learning who dreams of creating a general theory of learning, analogous to E=mc² for physics.

Richard Sutton

Author of a reinforcement learning book, which Peter Abbeel read before the resurgence of deep learning.

Chelsea Finn

Led research at Berkeley in meta-learning for demonstrations, enabling robots to learn from human actions by translating them into the robot's own motor control.

Roger Federer

A professional tennis player mentioned as a benchmark for robot capabilities in tennis.

Jeff Bezos

Organized a Mars event where SpotMini robots were showcased.

Steven Pinker

Author of 'Better Angels of Our Nature,' discussed in the context of historical trends towards decreased violence and increased cooperation among humans, relevant to AI policy.

Companies

DeepMind

Recognized for impressive results in Unreal Engine, where agents learned to navigate mazes without explicit objectives or reinforcement learning, demonstrating advanced learning capabilities.

OpenAI

Associated with Paul Christiana's work on teaching robots through comparative feedback, specifically mentioning the Hopper robot.

Boston Dynamics

A robotics company whose robots, like SpotMini, are highlighted for their impressive physical capabilities and parkour-like movements.

Google

Mentioned for its recent results in large language models that are learned for prediction and then reused for other tasks, showcasing transfer learning.

Books

Better Angels of Our Nature

A book by Steven Pinker that discusses the historical decrease in violence and increasing cooperation, relevant to understanding human behavior and AI policies.

Concepts

Reinforcement Learning

The core machine learning paradigm discussed, focusing on how systems learn through trial and error and sparse rewards, its challenges, and potential future directions.

Deep Reinforcement Learning

An area of research focused on making robots understand and interact with the world, combining deep learning with reinforcement learning.