Key Moments
Robots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna Robotics
Latent Space PodcastKey Moments
Dyna Robotics has developed a "world-action model" that allows robots to learn new tasks in under an hour and predict physical outcomes, but widespread deployment hinges on demonstrating significant value to clients beyond mere novelty.
Key Insights
Dyna 2, their world-action model, is trained on over one million hours of human video data, enabling robots to predict future physical states and generate actions.
Robots can learn new tasks, such as opening a refrigerator or folding a napkin, with as little as 45 minutes of demonstration data.
The company's robots are already folding thousands of napkins daily for clients like Din Tai Fung, indicating real-world production beyond typical demos.
Dyna Robotics emphasizes the difficulty of convincing clients of a robot's value, distinguishing their approach from companies solely focused on experiments.
The core architecture of Dyna 2 utilizes a diffusion transformer, combining the sequence modeling capabilities of transformers with the generative power of diffusion models for precise motion generation.
The company is actively seeking data partnerships, highlighting the critical role of high-quality data, including human-centric and robot-specific datasets, for advancing physical AI.
The Dyna 2 world-action model predicts physical futures
Dyna Robotics has introduced Dyna 2, a "world-action model" trained on over a million hours of human video data. This model is designed to predict the future state of the physical world and, crucially, generate actions for robots to execute. The core idea is to mimic human cognition, where we implicitly understand how the world will evolve based on our actions. By integrating this predictive capability into robot control, Dyna 2 aims to create more robust, generalizable, and adaptable robots that can perform in novel environments and scenarios.
Efficient learning from demonstrations
A key capability demonstrated by Dyna Robotics is the ability of their robots to learn new tasks rapidly. For instance, tasks like opening a refrigerator or folding a napkin can be mastered with as little as 45 minutes of demonstration data. This significantly reduces the training time and complexity typically associated with robotic skill acquisition, making robots more practical for real-world deployment where quick adaptation to new tasks is essential.
Real-world deployment and client value
Unlike many robotics companies that focus on impressive demos or limited trials, Dyna Robotics emphasizes the challenge and importance of proving tangible value to clients. Their robots are currently folding thousands of napkins daily for clients like Din Tai Fung, which has become their largest customer by scale. The company's success lies in convincing clients that the robots add sufficient value to justify widespread deployment across multiple locations, a hurdle many in the industry have yet to overcome.
The architecture of prediction and action
The input for Dyna 2 includes visual data from robot cameras and joint encoder data, along with linguistic instructions. The output consists of executable actions for the robot and, uniquely, a prediction of future video or world state. This future prediction is then used to guide the robot's actions, creating a continuous loop of perception, prediction, and action. The underlying architecture is a diffusion transformer, which leverages the sequence modeling power of transformers and the precise generative capabilities of diffusion models. This combination is particularly effective for generating smooth, high-fidelity robot movements, analogous to generating sharp images.
A foundational model for general-purpose robots
Dyna Robotics' approach centers on building a powerful, general-purpose foundational model trained on extensive data. The philosophy mirrors human learning: a strong general foundation allows for rapid adaptation to specific tasks with minimal fine-tuning. By training on a million hours of human data, the model gains a broad understanding of physical interactions. This allows robots to perform diverse tasks, as seen in office demonstrations, with simple instructions and a few examples, highlighting the potential for a single base model to handle numerous applications.
The critical role of data acquisition
In robotics, unlike language modeling with the internet, data is a significant bottleneck. Dyna Robotics collects data internally through its deployed robots and acknowledges the need to acquire data externally, either through purchase or partnership. They emphasize that human-centric data is invaluable for pre-training due to its diversity, providing a robust understanding of physical interactions. However, they also recognize the gap between human actions and robot execution, necessitating adaptation with robot-specific data, which is more challenging and costly to collect.
Iterative hardware and software development
Dyna Robotics adopts an iterative approach to both hardware and software development, recognizing that the physical embodiment of intelligence is complex and costly. While they leverage mature technologies like LiDAR and cameras, they develop critical components like actuators and robot kinematics in-house to ensure optimal performance and alignment with their AI models. This integrated approach allows for end-to-end optimization, from motor control to system-level deployment, crucial for achieving real-world robotic capabilities.
The future vision: ubiquitous, adaptable robots
The company's long-term vision is to integrate robots and AI into the physical world, deploying millions of robots across various industries. They aim for robots that can seamlessly perform complex, multi-step tasks, acting as true employees rather than just automated machines. This requires solving challenges in generalization, full-body control, rapid adaptation, language understanding, and dexterity. Dyna Robotics believes that by focusing on deploying robots in real-world settings and solving hard problems early, they can accelerate progress towards this goal.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Studies Cited
●Concepts
●People Referenced
Robotics Development: Key Principles
Practical takeaways from this episode
Do This
Avoid This
Common Questions
Dyna-2 is Dyna Robotics' global motion model, the first trained on over a million hours of data. Unlike models focused solely on video or language, Dyna-2 predicts future world states and generates physical actions for robots, making it more robust and generalizable.
Topics
Mentioned in this video
Dyna-2 is Dyna Robotics' global motion model, designed to predict future world states and generate physical actions for robots. It is trained on over a million hours of data and utilizes a diffusion transformer architecture.
The core architecture behind Dyna-2, combining the sequence modeling capabilities of transformers with the generative power of diffusion models for precise physical actions.
A large language model used in the 'Eureka' project to generate environments and reward functions for reinforcement learning in simulated robotics tasks.
The name of the paper by Richard Sutton that laid the groundwork for self-improving agents in a world model, influencing the naming of Dyna Robotics.
A large language model mentioned as an example of AI that has advanced from simple auto-completion to complex task execution through agentic systems.
An advanced AI model developed by Google DeepMind, explored by Jason Ma for its potential applications in robotics.
An AI program that mastered the game of Go, profoundly influencing Jason Ma's interest in artificial intelligence and self-improving systems.
A robotics company developing a general-purpose robot with a focus on a universal motion model (Dyna-2) and integrated hardware/software solutions.
A technology company where Jason Ma previously worked, contributing to projects involving AI models and robotics, including early work on the 'Eureka' project.
A subsidiary of Google where Jason Ma worked, focusing on advanced AI models like Gemini and their application in robotics.
A company mentioned as an example of others developing world models for robotics, adapting visual world models for their applications.
The company founded by Jason Ma, focusing on building a general-purpose robot platform and motion model to integrate AI into the physical world.
A client of Dyna Robotics where their robots are used for folding laundry, such as towels, showcasing large-scale deployment in hospitality.
A high-end restaurant chain and a major client of Dyna Robotics, where their robots fold thousands of napkins daily with high precision, demonstrating the value of general-purpose robots in demanding applications.
Robots developed by Dyna Robotics that are used by clients like Genji for tasks such as folding thousands of napkins daily. They represent a large-scale deployment for the company.
The latest iteration of Dyna Robotics' robot, featuring an auto-control belt and designed for end-to-end workflow automation in real-world industrial environments, such as a hotel laundry room.
A robot platform that some companies choose to deploy models on, as opposed to building their entire hardware system.
A specialized version of diffusion models that simplifies the conversion process, reducing the number of steps required and making it more efficient, particularly for motion and robotics.
A machine learning paradigm focused on agents that learn to improve themselves within an environment, a key area of research for Jason Ma and central to robotic control.
A leading robotics research laboratory at the University of Pennsylvania, where Jason Ma conducted his doctoral research on reinforcement learning and robotics.
The university where Jason Ma pursued his Ph.D., specializing in reinforcement learning and robotics, and where he was involved with the GRASP Lab.
More from Latent Space
View all 263 summaries
29 minHow to kill your SaaS: what we learned replacing a $43K/yr tool with AI agents
33 minWhy AlphaFold Didn't Solve Protein Folding — Pushmeet Kohli, Google DeepMind & Sal Candido, Biohub
58 minWhich GPU Clouds Are Actually Good? | ClusterMAX 3.0
104 minRecursive Language Models — Alex Zhang, MIT PhD
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free