Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71

Vapnik believes intelligence involves a 'world of pure ideas' (predicates) that combine with reality to create 'invariants' (specific features of objects). Understanding these abstract ideas and their projections onto specific examples (e.g., in music or images) is central to intelligence.

Vapnik suggests symmetry is a universal predicate important for understanding images. He proposes different types and degrees of symmetry (horizontal, vertical, diagonal, anti-symmetry) can be measured and used to create invariants, which help in tasks like digit recognition by reducing the set of admissible functions.

The challenge is to achieve state-of-the-art error rates in handwritten digit recognition (like MNIST) using significantly fewer examples (e.g., 100 times less than 60,000). This would require the discovery and application of powerful, universal predicates.

According to Vapnik, there's an infinite number of possible predicates. Discovering "good" predicates—those that significantly reduce the set of admissible functions and thus require less training data—is a difficult, often human, endeavor akin to a physicist finding contradictions in existing theories.

Convolutional networks are seen as implicitly using a single predicate: translation invariance. Vapnik argues that while deep learning works, it currently relies on heuristics. He suggests a more principled approach would define admissible sets of functions based on explicitly understood invariants/predicates.

Uniform convergence, which ensures that functions converge simultaneously across a set of functions, is considered the most powerful idea. It's necessary to guarantee learning because it prevents overfitting when selecting a function from a broad set.

Philosophy, particularly the abstract understanding of life from thinkers like Plato, defines the high-level concepts and ideas that guide Vapnik's work. His mathematical concepts, like predicates and weak convergence, are implementations of this philosophical understanding of how to approach learning.

Vapnik believes that critics describing music, art, or poetry use high-level predicates that capture essential ideas. By analyzing how these critics describe various forms of art, common universal predicates might be identified and then applied to visual recognition tasks, like understanding digits.

Privileged information refers to having descriptions of data in two 'languages'—for example, an image of a digit and a poetic description of that digit. Vapnik mentions an algorithm where using such dual-language descriptions leads to significantly better performance in learning tasks.