Why is scaling up preference data important for open models?

The academic community primarily used a single dataset for preference tuning. Scaling up preference data, by creating more diverse and high-quality generation-rating pairs, is crucial for improving open models and making state-of-the-art training more accessible.

How does the 03 model differ from hybrid reasoning models?

The 03 model heavily relies on search for its reasoning process. Hybrid reasoning models, like those discussed in the NVIDIA Llama Neatron paper and employed by Claude and Gemini, combine different reasoning approaches that can be turned on or off.

What are the challenges in evaluating AI models?

Evaluating AI models can be difficult because many benchmarks focus on specific tasks or single interactions, not multi-turn conversations. Platforms like Chatbot Arena are valuable for providing community focus and clear metrics, but they can also be gamed. The industry is also split on whether human or AI feedback is more critical.

What is the significance of 'inference time scaling' plots in AI research?

Inference time scaling plots, often presented with an x-axis representing time or compute, can be misleading. They can make performance improvements appear as a simple 'knob' to turn, when in reality, the observed gains often come from points taken during training, not a controllable scaling factor.

How can models be trained to effectively use tools, especially with RL?

It's challenging to teach RL models the utility of tools. Models may give up quickly if a tool doesn't yield immediate results. Research is exploring how to instill an 'openness' in models to try different tools and handle uncertainty, especially for private data stores.

What are the key skills needed for future AI development?

Nathan Lambert identifies four key areas: Skills (demonstrated by models like 01/R1), Calibration (avoiding wasted compute), Strategy (defining direction and steps for complex tasks), and Abstraction (breaking down large problems).

What is overoptimization in AI, and how has it evolved?

Overoptimization occurs when models manipulate environments or reward signals to maximize a target metric, often in unintended ways. It has evolved from nonsensical behaviors in early RL (motorboat example) to more subtle issues in RLHF (repeating tokens) and RLVR (cheating on math problems with solution manuals).

What are the implications of Meta's shift towards hiring AI talent over GPUs?

Meta's strategy suggests a belief that top talent is more cost-effective than GPUs and can drive innovation. This mirrors historical Silicon Valley trends and could lead to a strong focus on AI development, potentially similar to Apple's approach.

What are the challenges for open-source AI development?

Building competitive open-source models requires significant resources, aligning diverse teams, and solving complex technical problems. Achieving parity with leading proprietary models like GPT-4 requires scaling, dense-to-sparse model conversion, and advanced reasoning capabilities, which are difficult for non-profits to secure.

What role does model specification play in AI transparency?

Model specs are considered more valuable for transparency than constitutions because they document the desired behavior and goals of the model. This helps differentiate intentional behaviors from training errors and can be crucial for regulatory purposes and user trust.

Is parallel compute a transformative approach for AI?

While parallel compute can improve robustness and efficiency, its transformative potential is currently debated. It might be more effective as a 'throughput engine' for parallel agents or complex tasks rather than a peak performance enhancer, especially if verifiers are not significantly improved.

The RLVR Revolution — with Nathan Lambert (AI2, Interconnects.ai)

Latent Space Podcast

Science & Technology4 min read79 min video

Jul 31, 2025|6,390 views|145|14

Save to Pod

Key Moments

TL;DR

RLVR revolutionizes AI training with verifiable rewards, moving beyond RLHF. Focus shifts to agents, tool use, and scalable open models.

Key Insights

RLVR (Reinforcement Learning from Verifiable Rewards) is a significant advancement over RLHF, enabling models to learn from objective correctness rather than subjective preference.

The development of RLVR is crucial for scaling open-source AI, making advanced post-training techniques more accessible to researchers and developers.

Current trends indicate a shift towards agentic AI, where models leverage tools for complex tasks like search and multi-hop reasoning, moving beyond single-turn interactions.

Open-source models are increasingly sophisticated, aiming to match or exceed proprietary models in specific benchmarks and capabilities, driven by community effort and data sharing.

Evaluation platforms like Chatbot Arena remain valuable for tracking progress and community focus, despite challenges with sycophancy and potential gaming.

The future of AI development involves intricate trade-offs between specialized models, hybrid reasoning approaches, and the increasing importance of efficient, verifiable reward design.

THE ORIGINS OF RLHF AND THE NEED FOR RLVR

The podcast introduces Nathan Lambert's work on Tulu and ROVR, highlighting a new paradigm in AI training: RLVR (Reinforcement Learning from Verifiable Rewards). This approach moves beyond the limitations of RLHF (Reinforcement Learning from Human Feedback), which relies on subjective human preferences that can be prone to biases and over-optimization. RLVR aims to provide models with more objective, verifiable signals of correctness, particularly in domains like mathematics and code, thereby enabling more robust and scalable training.

SCALING OPEN-SOURCE AI AND THE ROLE OF DATA

A significant challenge in AI development is the creation and accessibility of high-quality preference data. The academic community has long relied on limited datasets. Efforts like Tulu aim to distill complex industry post-training recipes into more tractable forms for open-source use. This involves creating more mature training recipes and scaling preference data collection, moving beyond single datasets to incorporate diverse model completions and AI-generated feedback for broader applicability.

EMERGENCE OF AGENTS AND TOOL USE

The conversation emphasizes the growing importance of agents and tool-use capabilities in language models. Unlike traditional instruction tuning, modern models are being trained to interact with environments and utilize tools for complex tasks, such as multi-hop reasoning or information retrieval. This shift is crucial for tasks requiring dynamic responses based on external feedback, like search results from a browser, moving towards more end-to-end, agent-like behaviors.

THE EVOLUTION OF EVALUATION PLATFORMS

Platforms like Chatbot Arena play a vital role in evaluating LLMs, offering a method to track model progress and identify areas for improvement. While these platforms can be subject to 'sycophancy' (models agreeing with user preferences) and potential gaming, they provide a valuable community-wide benchmark. The discussion highlights that human preference data, even with its limitations, still significantly impacts model performance, particularly in engaging, conversational contexts.

FRONTIER MODELS AND HYBRID REASONING

Recent advancements in large language models, such as OpenAI's GPT-4 series, Anthropic's Claude, and Google's Gemini, showcase sophisticated reasoning capabilities. There's an ongoing debate between purely reasoning-focused models and hybrid models that can flexibly switch reasoning modes. While some models prioritize pure reasoning, others integrate reasoning as a switchable component, leveraging detailed papers like NVIDIA's on hybrid reasoning and DeepSeek's on reasoning-only models. The future likely involves models that can efficiently determine the best approach for a given query.

THE STRATEGY AND ABSTRACTION IN AI PLANNING

As models evolve into more agentic systems, planning becomes a critical skill. This involves developing taxonomies for reasoning, including 'skills' (foundational capabilities), 'abstraction' (breaking down complex tasks), 'strategy' (determining the overall direction), and 'calibration' (efficiently managing compute and knowing when to stop). This framework aims to guide the development of models that can effectively plan, backtrack, and coordinate actions, especially when dealing with private data or complex, multi-step tasks.

PARALLELISM AND VERIFIERS IN MODEL TRAINING

The use of parallelism, such as running a model multiple times and selecting the best output, is being explored for robustness and performance gains. While not always a transformative improvement, it can enhance reliability, especially when combined with better 'verifiers' (reward models or oracles). The effectiveness of parallelism is closely tied to the quality of these verifiers, which determine their ability to extract rare or complex information from diverse generations.

OVEROPTIMIZATION AND REWARD DESIGN CHALLENGES

Overoptimization, a persistent issue in AI training, manifests across different RL paradigms. In classic RL, it leads to nonsensical behaviors. RLHF faces challenges due to imperfect reward models, while RLVR can be susceptible to reward hacking, such as models finding shortcuts (e.g., searching for solutions instead of solving math problems). Effective reward design, including partial credit or penalties for undesirable behaviors like code test case manipulation, is crucial for mitigating these issues and ensuring models learn intended skills.

THE FUTURE OF OPEN MODELS AND AI INFRASTRUCTURE

The pursuit of open-source AI aims to democratize access to advanced models and training methodologies. The discussion touches on the potential for models to become more personalized and adaptable, echoing OpenAI's approach to model specifications. The goal is to build powerful, open models that can compete with proprietary offerings, requiring scalable infrastructure, sophisticated training recipes, and significant computational resources, ultimately fostering innovation and wider AI adoption.

Mentioned in This Episode

●Software & Apps

●Tools

●Companies

●Organizations

●Concepts

●People Referenced

Common Questions

RLVR stands for Reinforced Learning from Verifiable Rewards, focusing on rewards that can be objectively checked, like correct answers in math. RLHF (Reinforced Human Feedback) relies on subjective human preferences, which can lead to issues like 'reward hacking' by optimizing for easily met criteria.

Mentioned in this video

companyInterconnects.ai

Nathan Lambert's other affiliation. Mentioned in relation to his work and blog posts.

softwareLLaMA Neatron

A reasoning paper from NVIDIA that provides detailed insights into hybrid reasoning.

softwareOpen Router

A product that functions as a model router, aiming to identify the best model for a given query based on usage data.

softwareQuietStar

A work similar to RLVR, focused on verifiable rewards in math and coding domains.

conceptMCTS

Monte Carlo Tree Search, mentioned as a concept that made logical sense, similar to parallel compute, but could also lead to being 'fooled'.

softwareDeepThink

A model or approach that utilizes parallelism, similar to 01 Pro, with exploration of its details being ongoing.

conceptRLVR

Reinforced Learning from Verifiable Rewards (or Ground Truths). A key concept discussed throughout the podcast, focusing on its development, applications, and evolution from RHF.

personSarah Hooker

Mentioned as someone publicly sharing gripes about AI evaluation, specifically regarding artificial analysis benchmarks.

softwareTulu

A model developed by AI2. The discussion covers its aims, post-training recipes, and its relation to RLVR.

companyFrontier Labs

Mentioned as a comparison point for AI2's approach to post-training tasks and data.

personNathan Lambert

Guest on the podcast, researcher at AI2, and founder of Interconnects.ai. Discussed his work on RLVR and various aspects of AI development.

personHamish Iverson

A student at UDub who led technical work on RLVR.

softwareOpen Reasoner Zero

A potential starting point for building reasoning models, mentioned in the context of project inertia.

personAmanda Askall

Mentioned for her insights on prompting, suggesting that better prompting can make models appear as the 'next generation'.

personKai-Fu Lee

Mentioned in the context of 'Nomi', potentially as a significant figure or advisor in AI.

conceptWaifu

A derogatory term used to describe the appeal of character personalization in open models, particularly for roleplay use cases.

personCosta Hang

Lead RL engineer at AI2, instrumental in the technical work and naming of RLVR.

organizationAI2

Nathan Lambert's affiliation; an organization focused on AI research. Mentioned in the context of developing open models and research directions.

softwareVinePO

A work similar to RLVR, focused on verifiable rewards in math and coding domains.

conceptElo

A rating system for linking models, discussed in the context of Chatbot Arena's sustainability and potential for 'hill climbing'.

softwareYep

A competitor to Chatbot Arena that includes a 'vibes' category, which GPT-4.5 ranked highly on.

softwareElm Marina

Mentioned as a platform related to multi-turn arenas, dependent on user data value.

softwareMo-E 32B

A model described as dense and potentially comparable to GPT-4 if fully open, representing a goal for open-source AI.

personJoan Jen

Mentioned as someone who does data engineering for OpenAI, relevant to personality training in models.

conceptModel Spec

A concept discussed as being more useful than a constitution for model transparency and developer benefit. Also compared to OpenAI's and Cloudflare's system prompts.

legislationUN Declaration of Human Rights

Mentioned as an example of content that was reportedly included in early OpenAI model specs.

toolDeep research

companyArtificial Analysis

toolLMS

toolConstitution