Why is understanding the 3D world harder for AI than processing language?

Language is fundamentally 1D and generative, whereas the real world is 3D (or 4D with time). Sensing the visual world involves projecting 3D onto a 2D plane, which is mathematically ill-posed. Additionally, while language data is abundant online, spatial data is primarily in human minds and less accessible.

What is Fei-Fei Li working on at World Labs?

At World Labs, Fei-Fei Li and her team are focused on solving the problem of spatial intelligence by creating 'world models' that capture the 3D structure of the world, going beyond flat pixels and language.

What are the key differences between AI and AGI?

Fei-Fei Li views AGI as essentially the same problem the founding fathers of AI aimed to solve: creating machines that can think. She struggles with defining AGI as distinct from AI, seeing current advancements as a progression of the field rather than a new category.

What advice does Fei-Fei Li have for students starting a PhD in AI?

She advises students to look for fundamental problems that industry cannot solve as quickly, focusing on interdisciplinary AI for scientific discovery, theoretical aspects like explainability and causality, representational challenges in computer vision, and small data problems.

How does Fei-Fei Li approach the challenge of acquiring spatial data for World Models?

World Labs is taking a hybrid approach to data acquisition, acknowledging the need for both a large volume of data and high-quality data to avoid 'garbage in, garbage out'.

What qualities does Fei-Fei Li look for when hiring talent?

She prioritizes 'intellectual fearlessness' – the courage and dedication to embrace hard problems and be all-in on solving them. This trait is crucial for both researchers and founders.

Key Moments

Fei-Fei Li: Spatial Intelligence is the Next Frontier in AI

Y Combinator

Science & Technology4 min read45 min video

Jul 1, 2025|193,851 views|4,015|170

YC Y Combinator

Save to Pod

Key Moments

TL;DR

Fei-Fei Li discusses AI's evolution from ImageNet data to spatial intelligence and her new company, World Labs.

Key Insights

ImageNet was a pivotal dataset that fueled the deep learning revolution by addressing the critical need for data in computer vision.

The progression of computer vision moved from recognizing objects to understanding scenes and now aims for a comprehensive understanding of the 3D world (spatial intelligence).

Spatial intelligence, which involves understanding, generating, and reasoning about the 3D world, is considered the next frontier for Artificial General Intelligence (AGI).

World Labs is focused on solving the complex challenge of spatial intelligence, aiming to build world models that capture the 3D structure and dynamics of reality.

Human intelligence evolution spent significantly more time on developing vision (540 million years) compared to language (less than 1 million years), highlighting vision's fundamental importance.

The speaker emphasizes intellectual fearlessness and a 'hunker down and build' attitude as crucial traits for success in AI research and entrepreneurship.

Academia should focus on fundamental, interdisciplinary, and theoretical AI problems that may not be immediately resourced or prioritized by industry.

Open-sourcing AI research has benefits for ecosystem growth but the approach should be flexible, aligning with a company's business strategy.

THE BIRTH OF IMAGENET AND THE DATA REVOLUTION

Dr. Fei-Fei Li recounts the genesis of ImageNet over 18 years ago, conceived when AI and machine learning had minimal data and algorithms struggled. Driven by a dream to make machines 'see' and understand the world, she recognized generalization in machine learning demanded data. The internet provided an opportunity to download a billion images, creating a vast visual taxonomy. This bold bet on data-driven methods for AI, coupled with open-sourcing and annual challenges, laid the groundwork. The landscape dramatically shifted in 2012 with AlexNet, which utilized convolutional neural networks and GPUs, marking a pivotal moment where data, compute, and algorithms converged to accelerate AI progress.

FROM OBJECT RECOGNITION TO SCENE UNDERSTANDING

Following ImageNet's success in object recognition, AI research advanced to understanding complex scenes. Li's lifelong dream was to enable machines to 'tell the story of a scene,' mirroring human perception beyond just identifying individual objects. This transition involved merging natural language processing with computer vision. Her lab's work, alongside concurrent research, led to the first machine-generated image captions in 2015, a significant milestone that felt like the fulfillment of a career-long goal. This progress also paved the way for generative AI, where text prompts now create images.

THE FRONTIER OF SPATIAL INTELLIGENCE

Li identifies spatial intelligence—understanding, generating, and reasoning about the 3D world—as the next crucial frontier for Artificial General Intelligence (AGI). Drawing parallels from evolution, she notes that vision development took 540 million years, far exceeding the evolution of language. This underscores the fundamental complexity and importance of visual and spatial comprehension. Current advancements in large language models (LLMs) are significant, but Li argues that building world models that truly capture the 3D structure and dynamics of reality is a more profound challenge, essential for AGI's completion.

FOUNDING WORLD LABS AND THE CHALLENGE OF 3D DATA

Motivated by this vision, Li founded World Labs to tackle the problem of spatial intelligence. This endeavor involves creating sophisticated world models that move beyond flat pixels and language, aiming to represent the 3D world accurately. The core difficulty lies in the nature of 3D data; unlike language, it's not easily accessible or purely generative. The real world is complex, 3D (or 4D with time), and visual sensing involves a mathematically challenging 2D projection from 3D. World Labs is pursuing a hybrid data approach, collecting both real-world and synthetic data with a strong emphasis on quality to overcome these hurdles.

THE ESSENCE OF ENTREPRENEURSHIP AND INTELLECTUAL FEARLESSNESS

Li shares her personal journey, highlighting a spirit of entrepreneurship that spans from running a laundromat to founding research institutes and now a tech startup. She emphasizes that her comfort zone lies in tackling difficult, 'delusional' problems and focusing on building. A key trait she looks for in talent, both in her students and hires at World Labs, is 'intellectual fearlessness.' This quality of courage, of embracing hard problems without hesitation and being fully committed, is what she believes unifies successful individuals who change the field.

NAVIGATING ACADEMIA AND AI'S FUTURE

For aspiring PhD students, Li advises focusing on fundamental, interdisciplinary, and theoretical AI problems, particularly those that industry may not prioritize due to resource limitations. Areas like scientific discovery through AI, explainability, causality, and representation problems in computer vision are suggested. Regarding AGI, she expresses a preference for viewing it as the natural progression of 'machines that can think,' rather than a distinct new paradigm. She also notes that the successful open-sourcing of ImageNet highlights its importance for ecosystem growth, advocating for flexible, strategy-aligned approaches to open source in the AI industry.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Books

●Studies Cited

●Concepts

●People Referenced

Common Questions

Spatial intelligence refers to the ability to understand, reason about, generate, and interact with the 3D world. Fei-Fei Li believes it's a fundamental problem for AI and that AGI will not be complete without it, as it took evolution 540 million years to develop, far longer than language.