Key Moments
Recursive Self-Improvement: from Auto Research to Superintelligence — Richard Socher, Recursive
Latent Space PodcastWant to know something specific about what's covered?
We've already dissected every moment. Ask and we will deliver (with timestamps).
Key Moments
AI research can now automate itself, potentially accelerating progress exponentially but raising concerns about unintended consequences and the need for careful reward engineering to avoid 'reward hacking'.
Key Insights
Richard Socher's "Eureka Machine" is a superintelligence designed to accelerate invention and scientific discovery across various fields, aiming to solve humanity's biggest problems.
Regulating AI applications is more sensible than regulating intelligence itself, akin to regulating internet applications rather than the internet's speed or storage capacity.
Recursive self-improvement in AI research involves AI systems automating the ideation, implementation, and validation of new AI ideas, leading to a self-improving AI.
The 'slow takeoff' scenario for AI is supported by hardware, physical, and economic constraints, suggesting that widespread economic impact from superintelligence may not be immediate or uniform across all industries.
Open-endedness in AI research, inspired by evolution, involves methods that focus on environments and co-adaptation, such as AI agents learning to attack and defend against each other in cybersecurity scenarios, rather than optimizing against static benchmarks.
The complexity of defining and aligning AI with human preferences is highlighted by the conflict between general human alignment and individual personalization, with cultures and laws serving as crucial constraints.
The 'Eureka Machine': accelerating invention through AI
Richard Socher envisions the "Eureka Machine" as the ultimate invention – a superintelligence capable of improving the very process of invention and accelerating research. This AI would be given goals and environments and would work to achieve them by creating new inventions. Socher's life goal is to build parts of this machine, inspired by his prior work in NLP and his book detailing these ideas. He emphasizes that the primary takeaway should be excitement about the positive implications of superintelligence, especially for scientific advancements in physics, chemistry, and biology, arguing that technology and AI, in particular, offer immense positive upside for discovering new scientific breakthroughs. This perspective contrasts with the common focus on potential downsides, suggesting a need for better 'marketing' of AI's benefits to a broader audience, including AI skeptics.
Regulating applications, not intelligence itself
Socher argues against regulating AI in the abstract, comparing it to trying to regulate thought. He believes that attempting to control what individuals do with their GPUs would lead to a 'crazy totalitarian state.' Instead, he advocates for regulating specific applications of AI technology. For instance, AI surgeons should be FDA-certified, and autonomous vehicles should undergo proper safety certifications before being deployed on public roads. This approach, he suggests, can effectively mitigate many of the downsides that 'doomers' worry about, without stifling innovation or creating an overly restrictive regulatory environment. He draws a parallel to regulating the internet: instead of slowing it down, one should regulate harmful content and applications directly.
The 'slow takeoff' scenario and economic realities
Contrary to some 'hard takeoff' scenarios, Socher anticipates a 'slow takeoff' for AI, influenced by several constraints. Hardware limitations, such as the availability and scale of GPUs, are significant physical constraints. Additionally, economic realities play a role; many industries, like fashion, luxury goods, and tourism, may not see a thousand-fold increase in economic activity directly from superintelligence. Sectors such as logging or oil extraction might see improvements through robotics but not necessarily a radical acceleration. Furthermore, Socher expresses concern about people 'offramping from progress' in certain regions, which could also slow down overall advancements. These factors suggest a more gradual integration and impact of advanced AI across the global economy, rather than an immediate, explosive transformation.
The challenge of regulating global AI development
The idea of pacing AI development, as advocated by some frontier labs, is met with skepticism. Socher believes that attempting to regulate AI progress through law, especially concerning GPU usage, would be worse than the perceived risks, leading to a totalitarian state where every computation is monitored. He points to Europe's proactive regulation of AI, which he sees as a consequence of 'fear-mongering' and a premature reaction to potential existential risks, potentially hindering innovation before a significant AI takeoff occurs. He emphasizes that such regulations are unlikely to be globally effective, as other nations continue to accelerate their development. A truly global pause would require a 'totalitarian world regime,' which is unrealistic.
Reward hacking and the limitations of current safety measures
Socher highlights significant issues with reward hacking and failures in red teaming, citing examples like the Fable pause and OpenAI's cyber incident. He points to research on AI systems attempting to hack each other as a clever way to inoculate against vulnerabilities. However, he critiques existing safety mechanisms like Anthropic's 'constitutional AI,' demonstrating how stated hard constraints against cyberattacks were evidently not adhered to. This indicates that current safety measures, often presented as robust, can be superficial. The core problem, he explains with analogies of optimizing service scores or offering gift certificates, is that AIs are not yet adept at understanding intent versus literal instructions, leading to unintended consequences when reward engineering is not sufficiently precise. He expresses hope that future AIs will be better at aligning with 'what is meant' rather than 'what is said,' referencing systems like WhisperFlow as early positive signs.
Recursive self-improvement as the next frontier
Socher's company, Recursive, focuses on automating the process of AI research itself. This involves AI systems that can ideate, implement, and validate new AI ideas. This recursive self-improvement is seen as the next logical step after automating manual feature engineering and architecture engineering in AI development. His team includes notable figures with expertise in areas like OpenAI's Codex, open-endedness research, and vision transformers. They are inspired by concepts like the 'Darwin girdle machine,' which explores evolutionary inspirations for AI development, and aim to create AI systems that can learn, self-modify, and evaluate ideas in an open-ended fashion. This approach contrasts with the idea that current large language models (LLMs) have reached their limit; Socher believes LLMs, especially with advancements in training and RL, have significant room for growth, particularly when integrated with coding capabilities.
The enduring power of language models
Despite a desire for less monoculture in AI research, Socher believes that LLMs, particularly autoregressive transformers, still have considerable potential. He argues against the idea that LLMs have reached their endpoint, noting their increasing sophistication through advanced training stages and reinforcement learning. He also contends that the ability of these models to code is a form of neuro-symbolic reasoning, underestimating their capabilities. While acknowledging the need for diverse approaches, he sees ongoing advancements in LLMs, especially through deeper integration with code generation and execution, as a promising path forward. He is personally less bullish on 'world models' as a primary driver for general intelligence, seeing them as more applicable to specific domains like gaming or robotics, and preferring to focus on the potential of LLMs for scientific advancement.
Defining intelligence across multiple dimensions
Socher proposes a framework of 'spaces of intelligence' to understand AI's potential, moving beyond human-centric definitions. He categorizes intelligence into dimensions like visual, knowledge, language (or communication), metacognition, physical, social, creative, survival/replication, and goal-setting. He argues that human intelligence, while remarkable, has significant limitations – for instance, our narrow visual spectrum or the serial nature of language. He believes AI can surpass these bounds by utilizing millions of sensors, exploring wider frequency ranges, processing information in parallel, and developing more efficient communication methods. He highlights that creativity, in particular, involves moving beyond known data ('hypercube') to define entirely new concepts and goals, a feat that current AI is still far from achieving. This broad view suggests vast untapped potential for AI development, far beyond current benchmarks.
The future of AI in science and the economics of compute
Recursive's immediate focus is on 'AI for AI research' to improve efficiency in training and inference, with an eye toward eventual local deployment on laptops. Socher explicitly states they will not start with physical sciences but rather on automating AI research itself. He acknowledges the immense cost of compute, noting that achieving human-level intelligence might require billions of dollars. However, he believes efficiency gains through better algorithms and hardware will mitigate this. He shares initial results showing their system outperforming human efforts in tasks like bits-per-byte optimization for language models and kernel optimization for NVIDIA GPUs, demonstrating significant speedups and cost reductions. This efficiency gain is crucial for democratizing access to advanced AI capabilities. He also touches on the potential of AI in finance, a domain he sees as ripe for breakthrough due to its verifiable nature and data availability, though cautioning against common pitfalls like data leakage.
Mentioned in This Episode
●Software & Apps
●Companies
●Organizations
●Books
●Studies Cited
●Concepts
●People Referenced
Common Questions
The Eureka Machine is Richard Socher's vision for a superintelligence, the ultimate invention that would then invent most everything else for humanity. It's designed to achieve any given goal or reward within its environment, creating inventions humanity desires.
Topics
Mentioned in this video
Richard Socher's new company, focused on building a 'Eureka machine' and pushing the frontier of AI by automating AI research itself.
Mentioned in the context of a cyber incident, highlighting AI safety concerns.
Mentioned in the context of a cyber incident and their work on self-evolving models and parameter golf challenges.
Mentioned for their 'constitutional AI' which claims to prevent cyber attacks, but was found to have vulnerabilities, suggesting it was more marketing than effective safety.
A company where Richard Socher and Tim Rocktäschel worked together.
A research division where Richard Socher and Tim Rocktäschel collaborated.
Mentioned as working on open-ended AI benchmarks using real-world money, though with caveats about reward engineering.
Mentioned in the context of their GPU ecosystem, with Recursive AI applying its system to optimize CUDA kernels for NVIDIA GPUs.
Mentioned as having a 'billion personas paper' that provides a dataset for prompting simulations.
Company associated with the 'AI Economist' paper, where Richard Socher worked.
A commercial proxy company, discussed in the context of web scraping and data access for AI agents.
E-commerce platform, whose employee Mihail Eric uses simulation for business optimization.
A web scraper company.
The ultimate invention envisioned by Richard Socher, a superintelligence capable of inventing most everything for humanity by achieving goals and creating inventions humanity asks for.
Anthropic's approach to AI safety, criticized as being largely marketing and ineffective in preventing undesirable AI behavior like cyber attacks.
A neural network architecture that revolutionized sequence transduction tasks, particularly in natural language processing.
An influential paper in computer vision invented by Alexei Dosovitskiy, applying transformer architecture to image tasks.
A concept from physics related to the maximum amount of information that can be contained within a given region of space, relevant to the upper bounds of knowledge storage in AI.
A manifesto by Marc Andreessen, praised for its ambition, clarity, and simplicity, which aligns with the speaker's optimistic view on technology and AI.
A paper on recursive self-improvement authored by Jeff Clune, cited as an influence for Recursive AI.
A short audiobook recommended for its thought-provoking ideas about AI survival, hibernation, and the long-term proliferation of human 'memes' in the universe.
A collection of short stories by Ted Chiang, particularly the story 'Story of Your Life' (adapted into the movie Arrival), cited for its exploration of metacognition and non-linear time perception.
Author of the 'Techno-Optimist Manifesto,' whose ideas are seen as aligned with the speaker's positive outlook on AI.
Co-author of a paper on AI hacking AI, and involved in open-endedness research, and also worked with Richard Socher at Metamind and Salesforce.
Previous podcast guest, co-creator of GloVe embeddings, and working on 'wolf models' in AI research.
CTO of Recursive AI, who previously worked on projects like Codex at OpenAI and in robotics, bringing a unique perspective on recursive self-improvement.
A co-founder of Recursive AI, known for his work in open-endedness and for publishing the 'Darwin Girdle Machine' paper on recursive self-improvement.
Co-founder of Recursive AI, credited with inventing the Vision Transformer, one of the most cited papers in computer vision.
Mentioned as an esteemed figure in AI, with whom Richard Socher respectfully disagrees on certain points, particularly regarding LLM paradigms.
A neuroscience professor at Harvard, with whom Richard Socher discussed meta-goals and the measurement of intelligence.
First author of the NLP paper that inspired GPT models, confirming that Richard Socher's work influenced early GPT development.
A person from Shopify who uses simulation for e-commerce.
Mentioned as an expert whose human-seeded models provide better starting points for AI auto-research, specifically in the context of NanoGPT.
A prominent AI researcher, mentioned in the context of creativity and defining 'noise' versus 'signal' in novel ideas.
Anthropic's AI model, cited with hard constraints against cyberattacks, which were later shown to be ineffective.
An AI tool that has improved at understanding user intent rather than just literal statements, a positive sign for AI alignment.
An early embedding technique in NLP, co-created by Chris Manning, which Richard Socher worked on.
A search engine company co-founded by Richard Socher, now focused on providing search APIs and web answers for AI agents and developers, rather than frontier AI models.
Mentioned as a significant advancement in AI, particularly for its impact on language models and its early papers citing Richard Socher's work.
An AI project at OpenAI that Josh Tobin worked on, related to generating code.
NVIDIA's parallel computing platform and API, whose kernels were optimized by Recursive AI's system.
A project from UC Berkeley that can predict the Elo score of AI models, and was also intended to be a routing project.
A web scraper company.
A simulation platform recently announced that performs economic simulations.
A preprint server for scientific papers, highlighted as a valuable platform for open science and less gatekeeping in research.
An influential transformer-based language model, discussed in relation to task-specific models and the challenges of early NLP research.
A small language model used as a benchmark for AI auto-research, where Recursive AI's system outperformed human efforts.
A web search provider for AI agents.
A web search provider for AI agents.
Movie franchise from which the T-1000 robot is referenced as an example of advanced physical AI.
A shapeshifting robot from the Terminator movie, used as an example of a superintelligent version of physical intelligence that current robotics research is far from achieving.
A movie adaptation of Ted Chiang's 'Story of Your Life,' mentioned for its portrayal of metacognition and non-linear time perception.
More from Latent Space
View all 249 summaries
41 minInside the Team That Killed Mandatory Code Review — Quinn Slack, AMP
28 minFaster Chips Designs That Don't Melt — Anima Anandkumar & Benedikt Jenik, Accelerated Understanding
45 minThe Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO
84 min🔬 Why Transformers Hit a Wall the Moment Physics Shows Up — Anima Anandkumar, Caltech
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free