Key Moments
Why Robotics Still Isn't Solved - But Could Be Soon | YC Paper Club
Y CombinatorWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
Robotics has been promised for years, but still struggles with the sim-to-real gap and embodied reasoning, making even simple tasks complex and expensive.
Key Insights
The 'sim-to-real gap' remains a major roadblock, where models trained in simulation fail to translate effectively to the physical world due to a lack of accurate physics modeling.
Robots lack the rich sensory input humans have, particularly in touch (epidermis), hindering their ability to build world models without vision, analogous to how humans can navigate a backpack by feel alone.
Memory is crucial for long-horizon robotics tasks, but current models without memory repeat mistakes indefinitely or perform tasks forever, while adding memory increases computational cost and can lead to performance degradation.
A key challenge is defining what robots should 'reason' about, with research showing that for some embodiments, reasoning about movement and gripper position is more effective than perceptual details, which can be distracting.
Training entirely in simulation using 'sim-to-real' reinforcement learning, like in the SimToolReal approach, can achieve extreme dexterity and generalization across novel tools and tasks by treating manipulation as goal-reaching, but relies heavily on accurate pose tracking.
The future of robotics applications lies in companies taking ownership of full business problems end-to-end, starting with teleoperation and off-the-shelf hardware, then iterating to incorporate AI, rather than solely focusing on foundational models.
The perennial promise and current reality of robotics
For the past decade, the 'next year' has been heralded as the year robotics would be 'solved.' Despite breakthroughs like AlphaGo and impressive demonstrations of policy training, like robots learning to walk through thousands of iterations or performing complex tasks such as making a sandwich or even shaving, a truly capable, generally available robot for everyday tasks remains elusive. While recent advancements like diffusion policies have shown promise for multi-step reasoning and tasks like watering plants or fixing bikes, the reality in mid-2026 is that we can still only pre-order many advanced robots, and 'Rosie the robot' is not yet a common household appliance. The current successes are largely confined to work cells, highlighting a significant gap between demonstrations and widespread deployment.
The foundational roadblocks: sim-to-real, action representation, and sensorimotor deficits
Several core challenges prevent robots from achieving human-level dexterity and adaptability. The 'sim-to-real' gap is a primary hurdle; simulation models often fail to respect real-world physics, leading to unrealistic behaviors like cars magically transforming into highways upon collision. Furthermore, learning effective action representations is still an 'unsolved thing' for rapid learning. Crucially, robots lack the sophisticated sensorimotor capabilities of humans. Human nerve endings provide rich information about force, friction, moisture, and temperature across the entire body, enabling intuitive world modeling through touch alone – something robots, with only limited fingertip sensors, cannot replicate. This deficit severely limits their ability to interact with and understand the physical world without constant visual input. Finally, 'embodiment drift' – the degradation of actuator performance over time due to wear and tear, corrosion, or environmental factors – requires constant retraining of models, making deployment costly and complex.
Memory for long-horizon tasks: necessity and computational burden
To tackle complex, long-horizon tasks like manufacturing or cooking a full meal, memory integration into robot policies is essential. Current policies often lack memory, meaning each observation is treated in isolation, leading to repetitive errors like endlessly washing dishes or burning food. Marcel's work on MAM (Multiscale Embodied Memory) addresses this by decomposing memory into short-term (dense frames for dexterity) and long-term (compressed language representations) components, feeding them into low-level and high-level policies respectively. While this approach significantly improves task completion and enables in-context adaptation, adding memory introduces challenges: effectiveness can be reduced due to distribution shifts and data scarcity, and efficiency suffers as increased context leads to more resource-intensive training and inference. The use of compressed text for long-term memory, while efficient, raises questions about what crucial information might be lost in translation compared to richer sensory data.
Reasoning for embodied agents: what to reason about and how
Milan Gennai's work focuses on 'embodied reasoning' for robots, akin to 'chain of thought' for LLMs, to improve action prediction in data-scarce robotics environments. The core challenge is identifying what constitutes 'good' embodied reasoning for specific embodiments. This involves overcoming the 'grounding problem' – the lack of an oracle source for reasoning data – and the 'verbosity problem' – determining which types of reasoning (e.g., planning, perceptual traces, gripper position) are most action-predictive without introducing excessive latency. Their approach, R&B encore (Refine and Bootstrap Embodiment Specific Chain of Thought Reasoning), uses a self-improving pre-training cycle. A reasoning proposer suggests potential reasoning types, and a reasoning validator scores them based on concision, non-triviality, and crucially, action-predictiveness. This self-supervised process generates synthetic, embodiment-aligned reasoning data that can retrain vision-action models (VAs), demonstrating that selective reasoning is more important than exhaustive reasoning, and that even valid reasoning isn't always useful for specific tasks or embodiments.
Sim-to-real for extreme dexterity: the SimToolReal approach
Tyler Lum presents SimToolReal, a method that trains a single policy entirely in simulation for dextrous manipulation, overcoming the limitations of teleoperation for complex tasks. By treating manipulation as goal-reaching, the policy learns to move arbitrary objects to desired poses, eliminating the need for task-specific reward engineering for each new tool or behavior. This goal-conditioned policy, trained with massively parallel RL, can then be deployed zero-shot in the real world to perform tasks like sweeping or using screwdrivers with remarkable speed (60 Hz) and dexterity. The key insight is that a human video demonstration can provide the sequence of goal poses, guiding a frozen policy without retraining. While powerful, SimToolReal's performance is sensitive to pose tracking accuracy, especially with small or occluded objects, and struggles with heavier tools. Interestingly, recovery behaviors, like regrasping a dropped object, emerged as a byproduct of including random forces in simulation, highlighting the importance of realistic simulated failure modes.
The rise of robotics application companies and the role of data infrastructure
Niko, co-founder of Rerun, advocates for a new category of 'robotics application companies' that own full business problems end-to-end, inspired by the success of SaaS. These companies focus on specific operational challenges (e.g., data center management, construction, warehouses) using teleoperation and off-the-shelf hardware initially, then iterating to integrate AI. The core idea is to start with a single customer problem, solve it scrappily, and learn through real-world deployment. This iterative approach allows for rapid learning of critical business requirements and failure modes that are impossible to anticipate in a lab. Effective data infrastructure, like Rerun's SDK and Hub, is crucial for managing the multimodal, episodic, and 3D nature of physical data, which differs significantly from traditional web data. This infrastructure facilitates rapid iteration, debugging, and scaling, enabling companies to build sustainable moats beyond just foundational AI models.
Optimizing world action models for real-time deployment
Bill from General Instinct discusses the evolution from Vision-Action Models (VAMs) to World Action Models (WAMs), which predict future states and kinematics. While WAMs like NVIDIA's Dreamer show promise, their computational demands (requiring multiple high-end GPUs and costing tens of thousands of dollars per model) make them economically unscalable for robotics. General Instinct focuses on optimizing WAMs for real-time inference. They employ techniques like distillation to reduce model size, separate diffusion transformers for video and action, and optimized sampling methods to decrease inference time from dozens of steps to just one or two, achieving a 50x speedup. Their infrastructure allows WAMs to run much faster and more efficiently, making them more viable for deployment on robots. The core problem they address is maintaining rich world representations efficiently, moving beyond pixel-level predictions to latent representations and using techniques like asymmetrical denoising to focus on high-detail areas.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Studies Cited
●People Referenced
Common Questions
Despite significant AI progress, robotics faces challenges in real-world modeling, handling deformable objects, precise sensory-motor control, and embodiment drift. These issues prevent robots from performing complex, long-horizon tasks reliably.
Topics
Mentioned in this video
An AI program developed by DeepMind that plays the game of Go. Its release is cited as an example of the recurring prediction that robotics would be solved soon after.
Mentioned as part of the progression of AI advancements that led to expectations of solved robotics, following AlphaGo.
A series of papers on world models, discussed in relation to the sim-to-real gap and the limitations of current physics modeling in robotics.
Multiscale Embodied Memory, a system developed by Marcel that adds memory to robot policies for long-horizon tasks.
Another type of robot policy mentioned as lacking memory, highlighting the need for memory in long-horizon tasks.
Vision Transformer, a model architecture used as a basis for Marcel's proposed short-term visual memory encoder.
Milan Gennai conducts research at AWS.
A vision-action model mentioned as an example of generalist policies in robotics demonstrations.
A vision language model used as a base for training vision-action models.
A vision language model mentioned as a base for training vision-action models.
A self-improving pre-training cycle for embodied reasoning VALAs, developed by Milan Gennai's team.
A framework developed by Tyler Lum's team that trains a single policy in simulation for diverse dexterous manipulation tasks.
A follow-up framework to SimToolReel, extending dexterous priors to precise contact-rich assembly tasks.
An example of a world action model that performs well on benchmarks but is computationally heavy.
A system developed by Liquin, using two encoders to predict future observations.
Mentioned as an example of a real-world location that simulated models might not accurately represent due to a lack of physics adherence.
A company building a unified data layer for physical AI, providing tools and infrastructure for working with robotics data.
Mentioned for previous research experience by founders of General Instinct in the area of world action models.
A company where Milan Gennai is currently working.
Mentioned in relation to Dreamer Zero and its performance benchmarks.
Authored a tech blog about training an image imagination model that predicts latent encodings for differences between adjacent frames.
Co-founder of General Instinct, a company focusing on infrastructure for running physical AI models.
Marcel's PhD advisor at Stanford, involved in the research on multiscale embodied memory.
A PhD student and presenter discussing embodied reasoning and self-supervised bootstrapping for action prediction.
Milan's PhD advisor at Stanford.
Milan's PhD advisor at Stanford.
A PhD student and presenter discussing SimToolReel and PlayToPerfect for dexterous manipulation.
Mentioned in relation to a meme about VALAs being dead and World Action Models being the future.
Associated with recent work on temporal difference encoding for image imagination models.
More from Y Combinator
View all 622 summaries
58 minMax Hodak: What Really Kills Deep Tech Startups?
57 minHow To Design In The Agent Era
43 minGarry Tan: "Personal AGI Is How You Stay Under Your Own Power"
37 minBuilding the First Data Centers in Space
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free