Sergey Levine: Robotics and Machine Learning | Lex Fridman Podcast #108

Sergey Levine views human intelligence as an 'iceberg of knowledge' built up over lifetimes through experience and adaptability, rather than solely inherited evolutionary traits. This suggests that learning plays a crucial role, offering hope for AI systems to acquire common sense through vast experience.

While robotics has pragmatic applications, Sergey Levine's deeper motivation is to use robotics as a tool to understand artificial intelligence itself. He believes that by studying how robots interact with the physical world, we can gain fundamental insights into the nature of intelligence, perception, and control.

Moravec's Paradox states that things easy for humans (like object manipulation or walking) are often hard for machines, while things hard for humans (like complex calculations) are easy for machines. Robotics highlights this paradox, suggesting that focusing on these discrepancies can reveal key missing insights in AI development.

Robotic grasping is difficult due to the enormous variety of objects and their properties beyond simple geometry, such as material flexibility, weight distribution, and potential for spillage. Different objects require vastly different strategies, and traditional inverse physics or geometry problems don't generalize well.

Sergey Levine suggests that common sense is an emergent property of interacting with and getting things done in a particular universe. AI systems, especially those in simulated environments (like image captioning), don't truly 'live' in our world. Forcing AI to deal with the messy complexity of our physical universe through robotics may compel them to acquire common sense.

Reinforcement learning can be seen as a generalization of supervised learning, relaxing the assumption of having explicit correct answers or i.i.d. data. While mathematically related, RL focuses on sequential decision-making to maximize utility through trial and error, whereas supervised learning is more about pattern recognition from labeled data.

Off-policy reinforcement learning involves learning from data collected by a different policy than the one being optimized. It's crucial for efficiently utilizing large prior datasets (like human driving data) without needing the agent to perform extensive, potentially risky, real-world exploration from scratch. A key challenge is knowing when to trust predictions for unseen actions.

The limits aren't just about neural network size but practical 'scaffolding issues' unique to the real world, such as breaking dishes during trial-and-error learning or defining accurate reward functions. Unlike simulators, real-world complexity introduces bottlenecks that current algorithms struggle with, highlighting the need for meta-learning and data reuse.

Levine argues that fundamental principles, like gravity, might be readily learned from data because their phenomena are constantly experienced. While human civilization has historically gotten stuck in local optima with scientific understanding, he believes machines interacting with the world might discover sufficient physical laws to act rationally without being explicitly programmed with them.

Unsupervised reinforcement learning aims to define a general objective, like minimizing a Bayesian measure of surprise, that encourages agents to explore and learn useful skills without explicit external rewards. This mechanism can lead to an emergent form of curiosity, where discovering new things is a natural consequence of optimizing for capability.

His dream is to build machines that can perpetually improve and get better the longer they exist in the world, unconstrained by human-built limits like simulators or fixed datasets. He envisions machines that can run up against the 'ceiling of the complexity of the universe,' always finding new things to learn and master.