How does Reinforcement Learning (RL) in robotics differ from its application in Language Models (LLMs)?

In robotics, RL involves an agent interacting with a dynamic environment, receiving states and rewards over time. In LLMs, the 'environment' is often a human providing feedback, and the 'state' (prompt) is usually discarded after a single completion, making it more akin to a 'Bandit problem' with limited interaction.

What are the core intellectual assumptions behind RLHF, and have they been debated?

RLHF relies on assumptions like the effectiveness of RL, the ability to quantify preferences (e.g., Von Neumann-Morgenstern utility theorem), and the aggregation of diverse human preferences (e.g., Bradley-Terry model). These assumptions have philosophical and economic roots, and concepts like 'Arrow's impossibility theorem' highlight challenges with aggregating preferences.

What is the difference between instruction tuning and RLHF, and which is more practical for startups?

Instruction tuning adapts a model to specific needs through hand-written samples and is generally more important and practical for most users and startups due to lower compute and straightforward loss functions. RLHF is more complex, involving preference data collection and optimization, and typically requires a larger team and more resources.

How is human preference data collected for RLHF, and what are its challenges?

Preference data is typically collected by presenting human labelers with multiple model completions (e.g., two responses) and asking them to choose the better one, often using a Likert scale. Challenges include the cost of labeling, human disagreement (model agreement rates are typically 65-75%), and ensuring the data reflects meaningful preferences beyond correctness and style.

What role does synthetic data play in RLHF, and what are the cost implications?

Synthetic data, often generated by more advanced AI models like GPT-4, is increasingly used because it can be significantly cheaper than human-labeled data (e.g., a tenth of a cent vs. $10 per data point). GPT-4 is often more accurate than humans at labeling preferences, though its use raises concerns about data contamination.

How does Constitutional AI work, and what is its goal?

Constitutional AI, developed by Anthropic, uses an AI model to critique and refine instruction responses based on a set of 'constitutional principles' (e.g., UN Declaration of Human Rights). It aims to align models with desired values by generating synthetic preference data from AI critiques, making the alignment process more scalable.

What is Direct Preference Optimization (DPO), and why is it gaining popularity?

DPO is a class of RLHF algorithms that directly derives an optimal reward function from preference data and an implicit optimal policy, using simple math based on log prob ratios. It's gaining popularity in open source due to its simpler implementation and effectiveness, with models like Zephyr and Tulu 2 showing success.

What are the current challenges in evaluating RLHF models?

Evaluating RLHF models is difficult due to the subjective nature of human preferences and the potential for models to 'overfit' to specific benchmarks. While leaderboards like Hugging Face's and academic benchmarks like AlpacaEval and Mt-bench provide some comparison, direct interaction with models is still essential to gauge qualitative improvements.

Key Moments

The Origin and Future of RLHF: the secret ingredient for ChatGPT - with Nathan Lambert

Latent Space Podcast

Science & Technology4 min read96 min video

Jan 11, 2024|2,348 views|52|2

Save to Pod

Key Moments

TL;DR

Explores RLHF's origins, sociology's influence, data tensions, and future research in AI.

Key Insights

Reinforcement Learning from Human Feedback (RLHF) has roots in diverse fields beyond computer science.

RLHF relies on fundamental assumptions about the measurability and aggregation of human preferences.

The practical implementation of RLHF is complex, involving trade-offs between human and synthetic data.

RLHF's effectiveness is debated, with critiques suggesting it may not always improve core capabilities.

Emerging directions like Direct Preference Optimization (DPO) aim to simplify and improve RLHF processes.

The future of RLHF may involve more AI-driven preference data generation and advanced alignment strategies.

THE EVOLVING LANDSCAPE OF REINFORCEMENT LEARNING

Reinforcement Learning (RL) has a rich history, initially applied to robotics and complex decision-making problems. The field draws from diverse backgrounds, including physics, engineering, and computer science, fostering a unique worldview focused on trial-and-error learning. While early RL focused on toy problems, the advent of deep learning and powerful tools like Transformers has enabled its scaling. However, the application of RL to language models presents unique challenges, with the concept of an 'environment' and 'state' often becoming abstract or even contrived compared to traditional RL settings. This evolution has led to a richer understanding of RL's core principles and its adaptation to new domains.

THE INTELLECTUAL FOUNDATIONS OF RLHF

The development of Reinforcement Learning from Human Feedback (RLHF) is deeply intertwined with concepts from various disciplines, extending far beyond computer science. It builds upon centuries of thought, including economic theories like the Von Neumann-Morgenstern utility theorem, which underpins utilitarianism and the quantification of preferences. Models like Bradley-Terry are fundamental for handling pairwise preferences. These theoretical underpinnings highlight a core assumption: human preferences are measurable and aggregable. This forms the bedrock for RLHF, though the existence and nature of such preferences remain debated in fields like economics and philosophy.

FROM DECISION-MAKING TO DEEP LEARNING: RLHF'S JOURNEY

Early RLHF approaches, dating back to around 2008, involved humans directly assigning scores or rewards to agent actions. A significant leap occurred in 2017 with the 'Deep Reinforcement Learning from Human Preferences' paper, which demonstrated that learning from pairwise human preferences could effectively solve RL tasks, sometimes outperforming traditional reward-based RL. This work highlighted the richness of trajectory-based human feedback compared to single-state rewards. While this approach proved powerful, the exact reasons for its success and its broader adoption in the following years remain areas of discussion, suggesting the deep insights derived from human comparative judgments were crucial.

INSTRUCT TUNING AS A PRECURSOR AND COMPLEMENT TO RLHF

Instruction tuning, a technique focused on adapting models to follow specific instructions, is often a prerequisite and a complementary process to RLHF, particularly in today's landscape. It's highly practical, enabling models to become comprehensible and follow user directives with relatively low compute and straightforward loss functions. This method is crucial for tasks like implementing system prompts or role-playing. While instruction tuning can achieve many desired outcomes, RLHF offers a different perspective, particularly for refining nuanced preferences and behaviors that are harder to codify in direct instructions. The interplay between these two techniques is key to developing advanced language models.

THE TECHNICAL MECHANICS OF RLHF AND PREFERENCE DATA

RLHF's core objective is to optimize a policy (the language model) to maximize a learned reward, often subject to constraints like KL divergence to prevent overfitting. This process requires preference data, typically collected through pairwise comparisons where humans select the better of two model outputs. This data trains a reward model, which then guides the RL optimization. While simple in concept, collecting high-quality preference data is challenging and expensive. Issues like annotator agreement, preference aggregation (e.g., Arrow's impossibility theorem), and the definition of 'preference' itself present significant hurdles.

CHALLENGES AND EMERGING DIRECTIONS IN RLHF

Despite its success, RLHF faces scalability issues and is not always a guaranteed path to improved capabilities; it has shown mixed results on standard benchmarks. Emerging directions aim to address these challenges. Direct Preference Optimization (DPO) offers a simpler, often more accessible alternative to traditional RL algorithms by directly optimizing a policy from preference data without an explicit reward model. Other advancements include rejection sampling and best-of-sampling, which leverage more inference compute to improve output quality. Constitutional AI explores using AI models guided by principles to generate preference data, addressing the limitations of human scaling and aiming for more robust alignment.

EVALUATION AND THE FUTURE OF MODEL DEVELOPMENT

Evaluating language models, especially after RLHF fine-tuning, remains a significant challenge. Reliance on automated benchmarks can lead to overfitting, and human interaction is crucial for truly understanding model behavior. The emergence of platforms like Chatbot Arena and academic leaderboards like AlpacaEval and MT-Bench provides valuable insights, though concerns about benchmark gaming persist. As models become more sophisticated, the focus is shifting towards more nuanced evaluation and understanding how different training methodologies impact model capabilities, safety, and alignment. The ongoing research into RLHF and its alternatives highlights the dynamic and rapidly evolving nature of LLM development.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Books

●Concepts

●People Referenced

LLaMA 2 Training Costs Comparison

Data extracted from this episode

Component	Estimated Cost Range (USD)
GPU Compute	$3-6 million
Preference Data (Human Labeling)	$6-8 million

Model Performance on Chatbot Arena Leaderboard

Data extracted from this episode

Model Version	ELO Score (Approximate)
GPT-4 March 14th	40+ ELO points higher than June 13th
GPT-4 June 13th	Lower than March 14th
GPT-4 Turbo	Notably ahead of other GPT-4 versions

Common Questions

RLHF aims to align large language models with human preferences and values. It allows models to generate more helpful, harmless, and honest responses by learning from human feedback, overcoming issues like repetition or undesirable text from vanilla pre-trained models.

Topics

AI & Machine Learning Programming & Software Science & Mathematics Open-source AI AI Alignment Deep Learning Large Language Models (LLMs)Reinforcement Learning From Human Feedback (RLHF)Instruction Tuning Preference Modeling Direct Preference Optimization (DPO)

Mentioned in this video

Locations

Berkeley

Where Nathan Lambert received his PhD.

People

Sam Altman

Briefly mentioned in the context of his firing, implying Q* was linked to it.

Paul Christiano

Author of the 'Deep Reinforced Learning from Human Preferences' paper (2017), which demonstrated that learning from human preferences could solve basic RL tasks more effectively.

Nishant Rajani

Former colleague of Nathan Lambert from Hugging Face, also starting a company in the RLHF space.

Luis Cristo

Mentioned as a person starting a company in the RLHF-as-a-service space.

Jim Fan

Mentioned as a person who observed the transition in decision-making methodologies, particularly with Decision Transformers.

John Schulman

Mentioned for his ICML talk on proxy objectives for RLHF, discussing issues like ChatGPT being verbose and having self-doubt or refusals.

Nathan Lambert

Guest on the podcast, holds a PhD from Berkeley, interned at FAIR and DeepMind, bootstrapped the RLHF team at Hugging Face, and is currently a research scientist at the Allen Institute for AI. He also maintains the blog 'Interconnects' and co-hosts the 'Retort' podcast.

Companies

ByteDance

Reported to have trained on GPT-4 data, leading to OpenAI revoking their access, an example of TOS enforcement.

Apple

Its terms of service are mentioned humorously as a potential source for Constitutional AI principles.

Stack Overflow

An example of a data source used for instruction tuning, formatting questions and answers for model training.

Mentioned as a platform where Nathan Lambert's professional bio is available.

Anthropic

Considered masters of RLHF, they developed proprietary techniques like Constitutional AI and use Likert scales for data collection. Mentioned in the context of their 'Constitution' for Claude.

Stability AI

A company that has released DPO models, contributing to the growing trend of DPO in the open-source space.

DeepMind

One of the labs where RLHF techniques were developed; Nathan Lambert interned here.

Hugging Face

Nathan Lambert bootstrapped the RLHF team here. Mentioned in the context of their leaderboard and open-source models.

Intel

A company that has released DPO models, indicating broader industry adoption of the method.

Software & Apps

Open Assistant

An open-source project that likely encountered challenges with bad answers in preference data when trying to implement RLHF.

LLaMA 2

Meta's language model, whose paper cited the effectiveness of RLHF and noted the surprise of NLP researchers at its utility, highlighting its cost and time effectiveness. Used rejection sampling for RLHF process.

GPT-2

An early OpenAI language model, mentioned in the context of the 'Learning to Summarize' experiment where initial RLHF techniques were applied.

DaVinci 003

One of OpenAI's older instruction models, used as a baseline for comparison in the AlpacaEval benchmark.

Claude

Anthropic's model, whose 'Constitution' tries to embed specific values into its behavior.

GPT-4

Considered more accurate than humans at labeling preferences (80% vs. 60-70%). Mentioned for its role in synthetic data generation and for providing feedback in evaluation benchmarks like Mt-bench.

Chatbot Arena

A platform by LM Cys for limited evaluation of language models, valuable for understanding user interaction, and showed GPT-4 Turbo's superior performance.

Alpaca

A language model often boosted by DPO, also part of a popular academic benchmark for evaluating chat capabilities, particularly comparing a candidate model to DaVinci 003.

Arc

One of the six evaluation tools on the Hugging Face leaderboard.

Interconnects

Nathan Lambert's blog, known for timely and opinion pieces, including popular posts on AI stress and job searches, and explanations of model training techniques like RLHF.

Mistral

A company that released a DPO model, acknowledging DPO as an expected path for model development.

Hugging Face Leaderboard

A platform for automatically evaluating and ranking open-source LLMs, providing a central place for comparisons but also susceptible to overfitting.

GPT-4 Turbo

A newer iteration of GPT-4, notably ahead of previous versions on the Chatbot Arena leaderboard, suggesting an effective 'bump' in model quality despite similar benchmark scores reported by OpenAI.

GPT-3

An earlier OpenAI language model.

InstructGPT

An OpenAI model that demonstrated the three-step RLHF process and produced 'incredibly pretty plots' of performance improvement. It tried to match the instruction tuning model to constrain the distribution.

A research topic Nathan Lambert 'opportunistically wrote about,' related to mathematical reasoning, suggesting it might have been a moderate benchmark bump.

Zephyr

An early successful RLHF model in the public domain, showing DPO success in open source with modest resources, influencing projects like Tulu 2.

MT-Bench

An academic leaderboard for evaluating multi-turn chat capabilities, where GPT-4 scores initial and follow-up responses.

ChatGPT

A language model whose creation was perceived as somewhat accidental. Used RLHF for its development and is mentioned as a benchmark for open-source models.

Vicuna

An LLM that demonstrated the power of instruction tuning on smaller models, bridging the gap from GPT-3-ish to GPT-3.5-ish performance in open source with minimal resources.

Tulu 2

An open-source DPO model released by the Allen Institute for AI, trained at a 70 billion parameter scale using a Zephyr recipe on TPUs. It achieved good benchmark scores with minimal parameter tuning.

Concepts

Von Neumann-Morgenstern utility theorem

An economic theory that forms the foundation of utilitarianism, crucial for quantifying and modeling preferences in RLHF.

KL Divergence

A distributional distance used as a constraint in RLHF objectives, acting as a guardrail to prevent overfitting to small datasets and maintaining model stability.

ELSaT

An evaluation metric for GPT-4's technical report, humorously called 'bogus' as it is less relevant to RLHF's core purpose.

Decision Transformer

A concept in AI related to using Transformers for decision-making, particularly in offline RL.

TruthfulQA

A benchmark that Ultra feedback data set boosts.

Constitutional AI

Anthropic's approach to alignment, where a second AI model evaluates a first model's outputs based on 'constitutional principles,' effectively modifying the RLHF setup with AI-provided critiques.

Weak-to-strong generalization

An OpenAI paper discussing how to make a weaker model (e.g., GPT-2) smarter by using a stronger one (e.g., GPT-4), relevant to superalignment and controlling future superintelligence.

Likert scale

A type of scale used in preference data collection, typically ranging from 1 to 8, where middle numbers represent ties and extreme numbers indicate strong preferences for one option.

Bradley-Terry model

A model from the 1950s used for pairwise preferences, which underlies how RLHF works by comparing two completions and determining which is better.

Utilitarianism

A philosophical theory that is foundational to the quantification of preferences, relevant to RLHF's aggregation of human feedback.

Studies & Research

Let's Verify Step by Step

A paper by OpenAI that uses best-of-N sampling, rewarding each step in chain-of-thought reasoning to make the problem more specific.

Hellaswag

An evaluation tool on the Hugging Face leaderboard.

Media

Retort

Nathan Lambert's new podcast with his friend Tom, focusing on critical perspectives on AI developments.

Organizations

FAIR

Where Nathan Lambert interned.

Allen Institute for AI

Nathan Lambert's current employer as a research scientist. They are working on releasing models and open-source pre-training language models, including Tulu 2.

Scale AI

A major company in human preference data labeling, responsible for supplying LLaMA 2's data. They also manage data collection workforce and handle disagreement in labels.

Legislation & Policy

UN Declaration of Human Rights

Mentioned as a potential source for the 'principles' in Constitutional AI, though the specific constitution is unspecified.