Key Moments
🔬 Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar, Caltech
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI models struggle with physics until neural operators are introduced, enabling weather forecasts on a consumer GPU that rival supercomputers and rival human doctors in diagnostic accuracy.
Key Insights
Neural operators can perform weather forecasting tens of thousands of times faster than traditional supercomputer-based models, running on a consumer-grade GPU.
TorchLean allows neural networks to be formalized and verified within Lean, enabling formal proofs of properties like certified robustness and bounds on outputs based on input perturbations.
Physics-informed neural networks (PINs) struggle with optimization, especially for time-dependent and turbulent problems, leading to the development of neural operators that leverage data alongside physical constraints.
Fourier Neural Operators (FNOs) offer a balance of efficiency and expressivity by utilizing the Fourier domain to efficiently capture non-local phenomena common in natural systems, with quasi-linear complexity.
ForecastNet 3, built on neural operators and incorporating Earth's spherical geometry, can perform both short-term weather and long-term climate modeling, a capability previously requiring separate models.
AI models, like those developed using neural operators, can accurately predict rare and extreme weather events, which was a surprise as they were expected to perform better on typical events.
AI's limitations with physical systems and the emergence of neural operators
Traditional AI, particularly transformers, often struggles when applied to the physical world due to insufficient data, brutal resolution requirements, and models that are not large enough. Anima Anandkumar highlights that AI's typical playbook breaks down when dealing with complex physical systems described by equations, such as those in weather, fusion, or materials science. The issue isn't a lack of data in some cases, like weather, but the inherent complexity and the decades of development in traditional physics-based modeling. The solution proposed is to build the structure of the world into the model itself, a concept that predates deep learning. This led to the development of neural operators, which, when applied to weather modeling using open-source data, produced forecasts that were not only accurate but also tens of thousands of times faster than traditional methods, running on a consumer GPU. This success challenged the notion that AI could not outperform established, physics-based models and significantly shifted perspectives in the field.
Formal verification of neural networks with TorchLean
Anandkumar's broader research program, 'AI and Science,' aims to build AI with guarantees of working in the physical world. This involves two key aspects: modeling the physical world with physics correctness, which is where neural operators come in, and verifying claims symbolically. TorchLean is a framework that bridges these two by allowing neural networks to be formalized and proven within Lean, a formal language. This enables rigorous verification of neural network properties, such as certified robustness. For instance, in safety-critical applications like controlling a nuclear reactor or a drone, TorchLean can provide bounds on how much the output will change given a certain perturbation in the input. This capability is crucial for ensuring the safety and stability of AI systems integrated into control loops, providing confidence in their deployment.
From physics-informed neural networks to data-driven neural operators
Physics-informed neural networks (PINs) attempt to solve differential equations by incorporating physical constraints directly into the neural network's loss function. However, PINs often face significant optimization challenges, especially for time-dependent or turbulent problems where chaotic behavior can emerge. Anandkumar explains that these landscapes can be 'hopeless' for optimization. Neural operators offer an alternative by leveraging abundant data. Instead of solving equations from scratch, they are trained on vast datasets, learning to predict solutions for different instances of equations. This data-driven approach, combined with the option to incorporate physical constraints, overcomes the limitations of PINs. At test time, the trained neural operator can quickly provide an answer without being stuck in a difficult optimization process, leading to significantly faster and more reliable predictions.
The multiscale power of Fourier neural operators
Neural operators are a generalization of standard neural networks where inputs and outputs are treated as continuous functions rather than fixed-sized tensors, allowing for arbitrary resolution. Fourier Neural Operators (FNOs) are a particularly successful architecture within this class. FNOs leverage the Fourier domain, which allows them to efficiently capture non-local phenomena—interactions that span large distances—a common characteristic in natural systems like fluid dynamics and material deformation. While derivatives in differential equations are local, their inverses (integrals) are non-local. FNOs handle this effectively due to the global nature of Fourier transforms. They strike a balance between computational efficiency (quasi-linear complexity) and expressivity, enabling global connections without the quadratic complexity of transformers. By adding nonlinearities and residual connections, inspired by other successful neural network architectures, FNOs can model complex, non-local phenomena.
Forecasting Earth's weather and climate with spherical neural operators
A prime application of neural operators is weather forecasting. Anandkumar's team developed ForecastNet, which uses Fourier neural operators. Initial versions treated the Earth as a rectangle, suitable for short-term predictions. However, for longer-term climate modeling, ForecastNet 3 incorporates the Earth's spherical geometry. This architectural change provides crucial stability, preventing the blow-up seen in rectangular projections when running simulations over extended periods. By training on global reanalysis data and leveraging the efficiency of neural operators, ForecastNet can predict weather with high accuracy and at speeds thousands of times faster than traditional models. This democratization of weather modeling allows smaller agencies to achieve similar fidelity to larger ones and has been adopted by major weather agencies like ECMWF.
Ensemble predictions and handling extreme events
ForecastNet's ability to perform ensemble predictions—generating multiple possible outcomes by introducing small variations in initial conditions—is vital for both weather and climate forecasting, especially for assessing risk associated with extreme events. Traditional models struggle with large ensembles due to high computational cost. The speed of AI models like ForecastNet allows for much larger ensembles, providing more robust probabilistic forecasts. Surprisingly, these models have shown a strong capability in predicting rare extreme events like hurricanes, which was not initially expected. Anandkumar suggests this might be because extreme events, while rare, often possess specific physical signatures that AI can learn from the underlying structure of the physical world, even with limited data.
Digital twins for fusion energy and beyond
The application of neural operators extends to complex scientific domains like fusion energy. By creating a 'digital twin' of plasma evolution in Tokamak reactors, AI can simulate magnetohydrodynamics (MHD) equations a million times faster than traditional simulations. This enables researchers to study plasma behavior, predict disruptive events (where the plasma collapses), and design control strategies to prevent them, aiming for sustainable fusion. This approach is being extended to other fusion concepts like stellarators and relies on incorporating domain-specific geometry and physics. The underlying principle is leveraging AI to accelerate scientific discovery and design processes that are currently limited by the expense and time of physical experiments or traditional simulations.
The evolving role of theory and the future of AI for science
Anandkumar's career trajectory mirrors the evolution of AI, from building theoretical foundations to leading large-scale industrial applications and now focusing on principled AI for science. She argues that while pure data-driven approaches are reaching saturation, the physical world, with its limited data and need for extrapolation, requires a return to principled design—incorporating physical constraints, architectural inductive biases, and algorithmic rigor. The ultimate vision is for foundation models of physics, capable of simulating, designing, and controlling across a range of phenomena and even discovering novel designs through inverse problems, moving beyond narrow surrogates. The open-source neural operator library is available to facilitate this transition, empowering domain experts to apply these methods to their specific problems.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Concepts
●People Referenced
Common Questions
Neural operators generalize neural networks by treating inputs and outputs as continuous functions, allowing for arbitrary resolution and flexibility. Unlike standard neural networks with fixed-size inputs/outputs, neural operators can handle multiscale data and adapt to different resolutions at inference time.
Topics
Mentioned in this video
A lower-dimensional representation of data used in machine learning, which Anandkumar finds relevant to her work on modeling the physical world.
Neural networks that incorporate physical laws and differential equations into their training process to solve scientific problems.
A framework that allows writing and verifying neural networks within Lean, enabling formal proofs and certified robustness.
A formal language used for verification of mathematical statements and neural networks.
An AI system for protein structure prediction, cited as an example of AI success driven by underlying physical constraints.
A neural network architecture that uses attention mechanisms, which can be computationally expensive for high-resolution physical data compared to Fourier Neural Operators.
An algorithm for certified robustness, implemented within the TorchLean framework.
A company where Anima Anandkumar helped found the cloud AI team.
A class of models that generalize neural networks to handle inputs and outputs as continuous functions, enabling arbitrary resolution and learning mappings between function spaces.
A popular deep learning framework that TorchLean emulates for defining neural network layers.
A specific architecture of neural operators that uses the Fourier transform to efficiently capture non-local phenomena and provides a good tradeoff between efficiency and expressivity.
An AI-based weather model developed using neural operators, known for its accuracy, speed, and ability to predict extreme weather events and climate.
More from Latent Space
View all 245 summaries
37 min⏭️ Forward Deployed: Voice AI on what works in 2026
48 minExo: Harnesses should see their own code and logs — Alex Krentsel
96 min🔬Biology Is Turning Into Software — Matt McPartlon and Neil Patil, Chai Discovery
103 minThe Inference Frontier: 10x Faster Models to Self-Optimizing AI — Philip Kiely & Ali Taha, Baseten
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free