What is the role of math in understanding reality, according to Vladimir Vapnik?

Vapnik believes that by carefully examining the mathematical equations that define phenomena, we can uncover simple, underlying principles of reality. He sees mathematics not just as a tool but as a language that can reveal profound truths, much like poetry or music.

Why does Vapnik criticize deep learning?

Vapnik views deep learning as 'fantasy' and 'interpretation' rather than rigorous science. He argues that its architectures are arbitrary, it relies on massive datasets unnecessarily, and lacks a solid mathematical foundation compared to shallower networks and statistical learning theory.

What does Vapnik mean by 'admissible set of functions' and VC dimension?

An admissible set of functions is a collection of potential solutions that has a small VC dimension, meaning it's not overly diverse. This allows a learning model to generalize well from a limited amount of training data.

What are the limitations of current machine learning approaches?

Vapnik argues that current machine learning, especially deep learning, relies too heavily on vast amounts of data and often lacks a deep understanding of underlying principles. The real challenge lies in creating intelligent systems that can use data more efficiently by incorporating specific, useful predicates.

Can machines think like humans?

Vapnik questions the focus on imitation (like Turing's test) for intelligence. He suggests intelligence might not be solely internal but possibly external or collective, emphasizing the need to understand how humans learn and what makes a good teacher.

What is the most exciting problem in statistical learning for Vapnik?

He believes the ultimate learning story involves separating the statistical part from the intelligence part. The biggest challenge is understanding and generating the 'intelligence' needed to create effective predicates, akin to what a great teacher does.

Key Moments

Vladimir Vapnik: Statistical Learning | Lex Fridman Podcast #5

Lex Fridman

Science & Technology4 min read55 min video

Nov 16, 2018|104,322 views|2,280|128

Save to Pod

Key Moments

On this page

TL;DR

Vladimir Vapnik discusses the limits of current AI, the beauty of math, and the nature of learning and intelligence, contrasting instrumentalism with realism.

Key Insights

The distinction between instrumentalism (for prediction) and realism (for understanding) is crucial in machine learning.

Mathematics reveals fundamental, beautiful principles of reality, but human intuition often struggles to grasp them.

True intelligence in machine learning may lie in the human ability to devise relevant predicates, not just in vast datasets or complex architectures.

Deep learning's effectiveness is often based on interpretations rather than rigorous mathematical derivation, and it may not be the most efficient learning mechanism.

The development of effective mathematical models requires understanding the problem's underlying structure and 'invariants,' rather than just imitating human behavior.

The greatest open problem in AI is understanding the nature of intelligence itself, particularly how relevant predicates are generated by teachers and learners.

INSTRUMENTALISM VERSUS REALISM IN MACHINE LEARNING

Vapnik distinguishes between instrumentalism, focused on prediction, and realism, aiming for true understanding. In machine learning, instrumentalism drives current approaches, prioritizing finding rules for classification or prediction. However, Vapnik argues that for genuine understanding, the focus should be on learning conditional probabilities, a more realist perspective that seeks to comprehend the underlying mechanisms of nature or the problem domain.

THE UNREASONABLE EFFECTIVENESS AND ROLE OF MATHEMATICS

Drawing inspiration from Wigner's essay, Vapnik highlights the profound and often surprising ability of mathematics to uncover the simple, underlying principles of reality. He emphasizes that mathematical derivation, even from simple algebra, can lead to non-intuitive but beautiful insights, suggesting that mathematical structures hold inherent knowledge about the world, and careful analysis can reveal these principles far better than human fantasy or intuition alone.

THE LIMITATIONS OF HUMAN INTUITION AND THE POWER OF DERIVATION

Vapnik expresses skepticism about the richness of human intuition, viewing even moments of brilliance as best understood as putting correct axioms into action, refined by generations of scientific thought. He contrasts this with mathematical derivation, which, through rigorous steps, can uncover surprisingly simple yet profound ideas that might escape human imagination. This suggests that human intuition is often primitive and needs to be guided by logical mathematical processes to achieve deeper understanding.

REDNINGTHE ESSENCE OF INTELLIGENCE: PREDICATES AND TEACHERS

Vapnik posits that true intelligence, particularly in learning, lies in the human ability to devise relevant predicates—concepts that effectively narrow down possibilities. He uses the analogy of a teacher selecting meaningful phrases like 'swims like a duck' over irrelevant ones. This generative aspect of intelligence, finding the right descriptions, is distinct from merely processing vast amounts of data and represents a significant open problem that current machine learning largely overlooks.

CRITIQUE OF DEEP LEARNING AND THE SEARCH FOR INVARIANTS

Vapnik critiques deep learning as often being based on 'fantasy' and interpretation rather than rigorous mathematical insight, frequently relying on immense datasets. He argues for the importance of identifying and incorporating 'invariants'—properties that hold true across different situations—as central to efficient learning. These invariants, when mathematically integrated, can drastically reduce the need for data and lead to more robust theoretical understanding, contrasting with deep learning's often arbitrary architectures.

THE PROBLEM OF LEARNING AND THE VC DIMENSION

The complexity of learning, as described by Sample Complexity, hinges on the concept of VC-dimension. A small VC-dimension indicates a limited capacity or diversity within a set of admissible functions, allowing for effective learning with fewer observations. Vapnik emphasizes that the core challenge in statistical learning theory is not just selecting a function from a given set, but actively creating an admissible set of functions with small VC-dimension that also guarantees the presence of a 'good' function.

THE SEPARATION OF STATISTICAL LEARNING AND INTELLIGENCE

Vapnik proposes that progress in machine learning requires separating the statistical part of learning from the 'intelligence' part. While statistical methods can be well-defined and rigorously analyzed, the generation of meaningful predicates, or the 'intelligence' component, remains poorly understood. Understanding how teachers select effective predicates and how learners internalize them is crucial for advancing AI beyond current data-intensive methods, leading to more efficient and insightful learning systems.

THE NATURE OF INTELLIGENCE AND EXTERNAL CONNECTIONS

The conversation touches upon the nature of intelligence, questioning the notion that it resides solely within an individual. Vapnik suggests that intelligence might be a more distributed phenomenon, with ideas and discoveries emerging simultaneously in different places, hinting at a collective or external network. This perspective challenges the purely individualistic model of human thought and creativity, suggesting a broader interconnectedness in the generation of knowledge.

EFFICIENCY THROUGH INVARIANTS AND REDUCED DATA REQUIREMENTS

A key theme is the potential for dramatic efficiency gains in machine learning by focusing on invariants. Vapnik challenges deep learning approaches by posing that similar results can be achieved with orders of magnitude less training data if relevant invariants are incorporated. This highlights the power of understanding the problem's structural properties rather than relying on brute-force data processing, making learning more efficient and theoretically grounded.

THE PHILOSOPHY OF SCIENCE AND THE POETRY OF DISCOVERY

Vapnik reflects on the philosophical underpinnings of scientific inquiry, noting a sense of beauty and poetry in discovering fundamental truths, whether in mathematics, music (like Bach), or nature. He values honesty and rigor, seeking to connect findings to 'ground truth' rather than relying on personal interpretations or 'blahblahblah.' This pursuit of deep, lasting principles guides his research and his view of scientific progress.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Books

●Concepts

●People Referenced

Common Questions

Instrumentalism views scientific theories as tools for prediction and production, without necessarily claiming they accurately describe reality. Realism, on the other hand, believes scientific laws represent the true nature of how the world is structured.

Topics

Technology & Innovation Science & Mathematics AI & Machine Learning Mindset & Self-Improvement Neuroscience & the Brain Statistical Learning Theory Machine Learning Fundamentals VC Dimension Learning Mechanisms AI Limitations Philosophy Of Science

Mentioned in this video

Organizations

MIT

The institution hosting the course on artificial general intelligence as part of which this conversation is produced.

Institute of Control Sciences in Moscow

The institution where Vladimir Vapnik worked in the Soviet Union before moving to the United States.

Columbia University

The university where Vladimir Vapnik currently works as a professor.

Facebook Research

One of the companies where Vladimir Vapnik worked in the United States.

Concepts

P equals NP

A long-standing problem in computer science and complexity theory, mentioned briefly as an interesting question related to algorithm complexity.

uniform law of large numbers

A more stringent version of the law of large numbers, highlighted as crucial for formal statistics and useful for understanding learning with limited data.

Instrumentalism

A philosophical position that emphasizes the predictive power of theories, viewing them as tools for production or prediction rather than necessarily true descriptions of reality.

Einstein's special relativity

Mentioned as an example where imagination played a role in scientific discovery, which Vapnik contrasts with his view of derivation purely from mathematical equations.

Vapnik–Chervonenkis theory

A fundamental theory in statistical learning co-invented by Vladimir Vapnik, concerning the capacity of statistical classifiers.

Deep Learning

A form of machine learning that Vapnik largely dismisses as 'fantasy' and 'interpretation,' arguing it lacks rigorous mathematical foundation and relies on excessive data.

Realism

A philosophical position that aims to understand the fundamental nature of reality, believing that scientific laws represent true descriptions of how the world is.

People

Lev Vygotsky

Mentioned for his idea that intelligence may exist outside of the individual, influencing Vapnik's thoughts on collective intelligence and learning.

Carl Friedrich Gauss

Mentioned as one of the mathematicians who independently developed non-Euclidean geometry, illustrating simultaneous discoveries in science.

Vladimir Vapnik

Co-inventor of Support Vector Machines, Support Vector Clustering, and Vapnik–Chervonenkis theory, discussing foundational ideas in statistical learning and the limits of current AI approaches.

Eugene Wigner

Author of the paper 'The Unreasonable Effectiveness of Mathematics in the Natural Sciences,' which discussed the surprising applicability of mathematics to scientific phenomena.

Alan Turing

Considered for his question 'Can machines think?' and the concept of imitation as a benchmark for intelligence.

J. H. Conway

Mentioned in the context of information theory and the limit of one bit of information per example, relevant to discussing the efficiency of learning.

Albert Einstein

Mentioned for his famous quote 'God doesn't play dice,' which is used as a jumping-off point for discussing statistics and the nature of reality.

Johann Sebastian Bach

Mentioned for his structured and classic musical compositions, which Vapnik cites as an example of 'ground truth' that resonates with mathematical principles.

Companies

AT&T

One of the companies where Vladimir Vapnik worked in the United States.

NEC Labs

One of the companies where Vladimir Vapnik worked in the United States.

Software & Apps

Support Vector Machines

A foundational concept in statistical learning, co-invented by Vladimir Vapnik, used for classification and regression analysis.

AlphaGo

A system that beat the best human player at the game of Go, mentioned as an example of a system that works despite Vapnik's skepticism about deep learning's effectiveness.

Support Vector Clustering

A clustering algorithm co-invented by Vladimir Vapnik, extending the principles of Support Vector Machines.

Books

The Unreasonable Effectiveness of Mathematics in the Natural Sciences

A seminal paper by Eugene Wigner discussing the profound and often surprising applicability of mathematical concepts to describing the physical world.