Key Moments
Runway’s Bet Beyond Video: World Models, Robotics, and the Neural OS — Anastasis Germanidis
Latent Space PodcastKey Moments
Runway is evolving beyond AI video generation to build "world models" that simulate physics and power robots, aiming for a fully neural operating system. This shift could fundamentally change software development and human-computer interaction.
Key Insights
Runway's initial thesis was to rethink creative tools due to the rise of generative models, but research revealed their broader applicability beyond content creation.
The company made a significant investment in a thousand A100 GPUs in 2022 to train large-scale video models, aiming for a "latent diffusion/stable diffusion moment" for video.
Gen 2, initially a hackathon pipeline combining text-to-depth and depth-to-video, became a crucial step towards controllable video generation and inspired world model research.
Runway's 'interface world models' aim to replace traditional front-end code with a neural operating system that renders interfaces directly as pixels, controlled by clicks and prompts.
The company believes scaling video models will lead to simulating physics and human actions, a direction exemplified by their 'world models' research and benchmarks like Physics IQ.
Runway's approach to robotics involves pre-training general video models on vast third-person video data and then fine-tuning them for robotics tasks, requiring significantly less specialized data.
From creative tools to broader simulation
Runway's co-founder Anastasis Germanidis reflects on the company's evolution from its initial mission to create creative tools in response to early generative models. The initial thesis was that as generative models improved, the way content is created would need to fundamentally change. However, as the company delved deeper into the research behind these models, it became clear that their utility extended far beyond creative applications. This realization marked a pivotal shift, leading Runway to explore more ambitious goals, including the simulation of real-world physics and the development of sophisticated 'world models.' This expansion signifies a move from generating media to understanding and simulating the underlying mechanics of the world itself.
The genesis of large-scale video model training
In mid-2022, Runway made a significant strategic bet by investing in a cluster of a thousand A100 GPUs, a substantial undertaking for a Series B startup at the time. This investment was driven by the belief that scaling laws, similar to those observed in language models, would apply to image and video generation. The goal was to achieve a 'latent diffusion/stable diffusion moment' for video. Initially, the focus was on text-to-video generation, but the team found that video-to-video generation, which offers stronger conditioning, was an easier starting point. This led to the release of Gen 1 in January 2023, a depth-conditioned video model that predicted pixels from depth maps generated by a light diffusion model. This was followed by Gen 2, which initially utilized a two-stage pipeline, demonstrating a pragmatic approach to rapidly bringing advanced capabilities to users.
Controllability and the seed of world models
Runway's journey with models like Gen 2 highlighted the critical need for greater control beyond simple text-to-video prompts. Filmmakers and creatives desired more nuanced control over camera trajectory, object motion, and overall scene composition. This led to rapid development of conditioning mechanisms like 'motion brush' and 'camera control.' The camera control feature, in particular, was a revelation, allowing users to feel like they were navigating within a generated world rather than just creating a static video. This shift from content creation to world navigation served as a foundational seed for Runway's subsequent research into world models, emphasizing the predictive power of video generation as a path to simulating the world.
Towards a neural operating system
The ultimate vision for Runway, particularly with their 'interface world models,' is the creation of a fully neural operating system. This concept, explored by figures like Andrej Karpathy, suggests that interfaces themselves will become learnable components of applications. Instead of rigid, code-based interfaces, users will interact with applications through a system that renders pixels directly and learns from user interactions like clicks and drags. This approach moves beyond simply delivering a language model to delivering an entire application end-to-end, where the rendering and pixel generation are also learnable aspects. Such a system could revolutionize software development, making interfaces more intuitive, personalized, and dynamic, potentially breaking down the traditional silos of different applications into a more unified, AI-driven experience.
Simulating physics and the 'Physics IQ' benchmark
Runway's ambition extends to simulating the fundamental physics of the world. The core thesis is that as video generation models scale, they will inevitably gain the capacity to simulate physical phenomena, human actions, and dynamics with increasing accuracy. To measure this, Runway uses benchmarks like 'Physics IQ,' which assesses a model's performance on mechanical, fluid dynamics, and optical phenomena. Researchers provide an initial frame of a physical scenario, and the model predicts the subsequent frames. By observing predictable improvements in Physics IQ scores as compute and model scale increase, Runway gains confidence in their scaling hypothesis for video models. This approach suggests that understanding and generating realistic physical interactions is a natural outcome of scaling video prediction capabilities.
Robotics applications and real-to-sim correlation
Runway's generalized world models have found a significant application in robotics. Initially, robotics labs used Runway's video models to generate synthetic data for training robotic policies. However, it quickly became apparent that these models could function as simulators, allowing for online testing of robotic actions and closed-loop simulations. A key challenge in this domain is establishing 'real-to-sim' correlation, ensuring that actions taken in the simulation yield similar outcomes in the real world. Runway demonstrated strong correlation using their GWM1 model on benchmarks like Roborina, enabling faster and more scalable evaluation of robotic policies without extensive physical hardware. This work leverages the vast amount of third-person video data available to pre-train models, which can then be fine-tuned with far less specialized robotics data.
The 'Lucid Dream Test' and the future of interactive video
Germanidis proposes a 'Lucid Dream Test' as a thought experiment for evaluating the advancement of world models. The test envisions a future where real-time, interactive video models are so advanced that a person wearing a VR headset cannot distinguish between a live video pass-through of their surroundings and a fully generated, interactive environment. The ability to generate realistic counterfactuals – what happens if I take action A versus action B – is identified as a crucial differentiator between current video models and true world models, especially for applications like robotics where simulating failure is critical for improvement. This highlights the ongoing challenge of moving beyond generating plausible visuals to creating models that truly understand and can simulate the dynamics of the world.
The convergence of diffusion and autoregressive models
Runway's development of Gen 3, released after OpenAI's Sora, marked a significant leap, requiring them to adapt lessons learned from language models in a compressed timeframe. A key architectural shift was the adoption of diffusion transformers over earlier models like convolutional neural networks. This required building robust infrastructure for distributed training and model parallelism. The company also notes the trend towards 'omni models' that unify autoregressive text prediction with diffusion-based pixel generation. While currently only a few models like Gemini and Quillion explore this, Runway anticipates this unification. This approach allows for iterative reasoning, where generated outputs can be fed back into the model for refinement, creating a more dynamic and integrated generation process.
Mentioned in This Episode
●Products
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
Common Questions
Runway's initial thesis was that the increasing resolution and quality of generative models would necessitate a rethinking of creative tools. They believed that generative models could be used to create art in unexpected ways if the right tools were built around them for artists.
Topics
Mentioned in this video
The company co-founded by Anastasis Germanidis, focusing on generative AI for creative tools and world models.
Company that released the Pix2PixHD model; also a partner in the Cosmos Coalition.
AI research company with a mission to solve intelligence and use it to solve everything else.
AI research company, specifically mentioned for releasing Sora.
A brand of action camera used to describe the concept of egocentric data collection for robotics.
Company that released the ImageBind multimodal AI model.
An early image-to-image model released by NVIDIA that Runway used in early projects to generate photorealistic outputs from semantic maps.
A foundational generative model that marked a step-function change in AI image generation, leading to the development of Stable Diffusion.
A latent diffusion model that significantly advanced image generation, developed by researchers including Patrick Esser at Runway.
An open-source machine learning framework, mentioned in the context of early ML tooling.
OpenAI's advanced video generation model, which presented a significant challenge to Runway and led to the accelerated development of Gen-3.
Runway's video model released in 2024, developed rapidly in response to Sora, featuring a 10x increase in model scale and leveraging diffusion transformers.
Google's multimodal AI model, mentioned in the context of advanced AI capabilities.
Runway's latest base model, which forms the foundation for GWM1 and its robotics applications.
A physics engine and robotics simulator, mentioned as a traditional simulator.
Another AI model with a context limit of one minute, mentioned in comparison to GWM models.
A highly specialized AI for protein structure prediction, mentioned as an example of a fine-tuned architecture for specific scientific problems.
A generative AI model known for its image generation capabilities, mentioned as a significant advancement in 2022.
A cloud-based Jupyter notebook environment, used for early machine learning experiments.
Runway's first video generation model, released in January 2023, which was a depth-conditioned video-to-video model.
Runway's text-to-video model, initially implemented as a two-stage pipeline combining text-to-depth and depth-to-RGB, offering a significant improvement in video quality.
A generative AI model known for its detailed captions and prompt rewriting capabilities, which Sora built upon.
An early large language model by OpenAI, used as an analogy to describe the early, less coherent outputs of video generation models.
Runway's open-ended world exploration model, which can generate videos for a few minutes before error accumulation.
A world model built on top of Gen-4.5, specifically adapted for robotics by enabling autoregressive generation and action acceptance.
A traditional robotics simulator, mentioned as being effective for rigid objects but less so for complex interactions like cloth or slippery surfaces.
Runway's world model specifically adapted for robotics, starting from Gen 4.5 and fine-tuned on robotic data.
A multimodal AI model released by Meta that integrates six different modalities, demonstrating transfer learning between them.
A fine-tune of Stable Diffusion that became a capable music generator by training on spectrograms, illustrating modality transfer.
More from Latent Space
View all 255 summaries
92 min🔬Bio-security is an AI Arms Race - Eric Nguyen (CEO, Radical Numerics)
122 min🔬 Google's AI Scientist Started as an Attempt to Automate Kaggle — John Platt, Google Fellow
143 minWhy We Made Jev — Diogo Almeida, TypeSafe Co-founder & CEO
70 minA Worm With 302 Neurons Inspired Their Architecture — Ramin Hasani, Liquid AI
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free