Why are computational models essential for understanding the visual system?

Computational models are crucial due to the high dimensionality of visual inputs and the complex mechanisms involved. They help in formulating theories and principles to explain how the visual system performs its functions.

How does the structure of the retina facilitate understanding its function?

The retina has a relatively simple structure with three layers of cells, making it manageable for study. This modular organization, where each cell type covers a specific visual field like a mosaic, naturally translates into computational models.

What is the difference between the classical and the new approach to studying the retina?

The classical approach used artificial stimuli like moving bars. The new approach involves presenting natural stimuli to the retina in a dish and building accurate computational models to capture the responses, allowing for deeper insights.

Why are natural scenes challenging for understanding retinal function compared to artificial stimuli?

Natural scenes possess high spatial and temporal variance. The numerous types of bipolar cells, each with different receptive fields and sensitivities, mean that receptive fields are not static and change with every stimulus.

How are Convolutional Neural Networks (CNNs) related to the structure of the retina?

CNNs share fundamental building blocks with the early visual system. Spatial filters and thresholds in CNNs are analogous to the receptive fields and non-linearities found in retinal cells like bipolar cells.

What are the advantages of using intermediate-complexity models like minimal CNNs for retinal research?

These models leverage existing knowledge of the nervous system's structure and dynamics. They require less data and provide mechanistic interpretability compared to very large, data-hungry models.

How does the speaker's lab experiment with retinal recordings?

They place a dissected retina on a multi-electrode array and present visual stimuli, either artificial or natural. They then record the electrical activity of retinal ganglion cells to predict their firing rate responses.

What was the surprising discovery when analyzing the internal structure of the trained models?

The internal structure of the models, specifically the receptive fields of interneurons, closely resembled what is found in biological neurons, suggesting that the models captured key computational mechanisms.

How can computational models be used to discover new phenomena?

By fitting models to natural scene data and then testing them on phenomena they weren't explicitly trained on, researchers can observe if the model reproduces these phenomena, leading to new hypotheses about neural function.

What is the 'deleted stimulus response' and how did models help explain it?

This phenomenon occurs when the retina predicts a missing stimulus in a sequence, firing at the expected time. Models helped by showing how combinations of specific cell types and their firing dynamics contribute to this predictive response.

What is the difference between 'activations' and 'contributions' in neural network analysis?

Activations refer to the raw firing of neurons in a model, while contributions represent their actual causal influence on the output. Contributions are often more sparse and can differ significantly from activations.

How can mechanistic interpretability help us understand the difference between excitatory and inhibitory processes in neural circuits?

By analyzing individual pathways and their contributions, researchers can quantify the cancellation between excitation and inhibition. This allows for a deeper understanding of how specific circuits, like those involved in object motion detection, function.

Key Moments

Stanford EE292R AI and Data for Science | Autumn 2026 | Models of Computation in the Retina

Stanford Online

Education5 min read61 min video

Oct 17, 2026|5,697 views|58

Stanford Stanford Online AI Artificial Intelligence

Save to Pod

Key Moments

TL;DR

Deep learning models can now predict retinal cell responses to natural stimuli, revealing internal mechanisms that mimic biological neurons and leading to new scientific discoveries.

Key Insights

The retina, a relatively simple neural circuit with three layers and five basic cell types, serves as a powerful model for understanding neural computation due to its manageable complexity and ability to be isolated and studied ex vivo.

Modern deep learning models, specifically convolutional neural networks (CNNs), closely mirror the hierarchical structure and processing stages of the early visual system, from photoreceptors to ganglion cells.

Models trained on natural visual scenes can capture a wide range of retinal phenomena, including adaptation to statistics, feature detection, and predictive responses, even without explicit training for these specific functions.

Internal model mechanisms, when analyzed using methods like integrated gradients, show remarkable correspondence to real biological neurons, suggesting these models offer genuine mechanistic interpretability.

The computational models can be used to generate novel hypotheses about neural function, which were then experimentally validated, demonstrating their utility as tools for scientific discovery.

Analyzing neural network pathways reveals that complex phenomena like object motion detection heavily rely on inhibitory processes that cancel out excitation, a finding that would be difficult to uncover through traditional experimental methods alone.

Understanding neural computation: the retina as a model system

Professor Stephen Baccus begins by highlighting the fundamental challenge in neuroscience and artificial intelligence: understanding how complex networks compute. He posits that the retina, the early visual system, has been at the forefront of deciphering these computational processes due to its relatively simpler architecture. Comprising three layers of cells and processing visual input into signals for the optic nerve, the retina is amenable to study. Its computational functions include detection, adaptation, compression, pathway estimation, prediction, and classification of stimuli. The lecture aims to bridge the gap between observed function and underlying neural mechanisms, emphasizing the necessity of computational models for handling the high dimensionality of biological systems.

Bridging biological and artificial neural networks

Baccus draws a strong parallel between the functional components of the retina and the building blocks of convolutional neural networks (CNNs). He explains how spatial and spatio-temporal filtering, followed by thresholding, are core operations in both. The mosaic-like organization of retinal cells, where each receptive field covers a specific area, directly maps to the convolutional layers of CNNs. Multiple filter types in CNN layers correspond to the diverse types of bipolar cells and other interneurons in the retina. This architectural alignment allows for the development of mechanistically interpretable models where the internal workings of the artificial network can offer insights into the biological system. The goal is not just to predict neural responses but to understand the underlying computations.

The power of natural stimuli and deep learning models

A significant shift in approach has been the move from artificial stimuli (like flashing dots) to naturalistic visual scenes for training and testing models. While early attempts at modeling with linear-nonlinear (LN) models could capture basic receptive field properties, they failed to account for the complexity of natural scenes. Deep learning models, particularly CNNs, have proven far more effective. By training these models on extensive datasets of natural movies, researchers can achieve high accuracy in predicting the firing rates of retinal ganglion cells. This accuracy approaches the biological noise floor, indicating the models capture a significant portion of the neural computation. The choice of model complexity is crucial, balancing expressive power with interpretability, with 'minimal CNNs' serving as a sweet spot.

Internal model mechanisms mirror biological reality

Perhaps the most surprising finding is that the internal structure of these well-trained CNNs, despite being trained only to predict output from input, often mirrors the structure of biological neurons. Analyzing the 'receptive fields' of internal model units reveals center-surround organizations, varied temporal dynamics, and different sizes, remarkably similar to real retinal interneurons. Experiments have shown a significant correlation (around 0.5-0.6) between the responses of model neurons and actual biological neurons to identical stimuli, a correlation close to the biological variability between neurons. This suggests that these models are not just black boxes but contain interpretable internal components that reflect biological computations.

Discovering new phenomena and generating hypotheses

The interpretability of these models allows them to be used as tools for scientific discovery. Researchers have used them to investigate known phenomena, such as adaptation to contrast and predictive responses (like the 'omitted stimulus response'), and found that models trained on natural scenes can generalize and reproduce these effects without being explicitly trained for them. More excitingly, the models can reveal new phenomena. For instance, by analyzing a model trained on natural movies, researchers identified a specific type of object motion-sensitive cell, a phenomenon previously identified in retinal preparations but not observed with initial models trained on simple stimuli. When the models were retrained with stimuli containing differential motion, they successfully reproduced this. This iterative process of modeling, experimentation, and hypothesis generation is a powerful new paradigm.

Analyzing computational pathways and circuit dynamics

Going beyond simple neuron-level analysis, researchers are now using techniques like integrated gradients to dissect the contributions of individual neurons and pathways to the overall output. This reveals that many activated neurons in a network might not significantly contribute to the final output, a phenomenon observed in both biological and artificial systems. The analysis shows that complex computations, like object motion detection, often arise from the precise interplay of excitatory and inhibitory pathways, where inhibition plays a crucial role in shaping the final response. By examining these pathways, scientists can generate hypotheses about how specific circuit dynamics, such as the balance between excitation and inhibition, enable complex functions.

The role of prior knowledge and future directions

The lecture concludes by emphasizing the importance of inductive biases – incorporating prior knowledge about the nervous system into model design. While large, data-hungry models can capture complex relationships, models that leverage biological constraints require less data and offer greater interpretability. Future work involves refining these models, potentially incorporating known 'connectomes' (neural wiring diagrams) to further improve accuracy and mechanistic understanding. The retina, with its well-understood structure and function, continues to serve as a critical testbed for developing and validating these interpretable AI models, paving the way for a deeper understanding of both biological and artificial intelligence.

Mentioned in This Episode

●Software & Apps

●People Referenced

Common Questions

The retina's main function is to translate visual signals into information transmitted to the optic nerve. It achieves this by processing visual input through multiple layers of cells and synapses.

Topics

Neuroscience & the Brain Technology & Innovation Science & Mathematics Neural Networks Computational Neuroscience Retinal Circuits Machine Learning Interpretability Visual Processing Computational Models Artificial Intelligence For Science

Mentioned in this video

Software & Apps

Convolutional Neural Networks

Computational models gaining increasing traction, applied to retinal recordings.

ResNet

A type of computer vision classification network analyzed for its contribution and activation patterns.

People

Dan Yamin

Will be speaking about cortical models in a future seminar.

Suri Ganguli

Collaborator on research involving computational models of the retina.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free