Why is mathematical reasoning a good domain for train time scaling?

Mathematical reasoning benefits from train time scaling because it is a domain where outputs are often verifiable. This verifiability allows for the selection of correct answers or reasoning paths, which can then be fed back into the training process to improve the model.

What is the main challenge in manually annotating reasoning steps?

Manually annotating reasoning steps is very expensive because it requires humans to meticulously document the step-by-step logic for a large number of problems.

How does the STAR method generate more training data for reasoning?

STAR generates reasoning data by first having the model produce rationales for problems with known correct answers. If the model fails, it's given the correct answer and asked to explain backwards, thus bootstrapping more difficult reasoning examples.

What are the limitations of the STAR approach?

STAR assumes the final output's correctness is a proxy for reasoning quality, potentially overlooking errors in the reasoning chain. It also assumes the base model is strong enough to bootstrap from few-shot examples, and performance can plateau after several iterations.

Why was DeepSeek Math successful in mathematical reasoning?

DeepSeek Math succeeded by starting with a code-trained model (DeepSeek Coder), curating high-quality math content from the open web, and implementing a more memory-efficient RL algorithm called GRPO, which improved performance on benchmarks like MATH.

What is GRPO and how does it improve RL efficiency?

GRPO (Generalized Reward Policy Optimization) is an RL technique that reduces memory usage compared to traditional DPO by using a generalized advantage estimation, thus avoiding the need for a separate critic model and allowing scaling to larger models.

How does DAPO address limitations in RL for complex reasoning?

DAPO introduces techniques like asymmetric clipping to prevent exploration collapse, dynamic sampling to maintain an effective batch size with informative rewards, and token-level loss functions to control response length and improve training stability on harder problems.

What is the difference between improving 'majority at K' and 'pass at K' accuracy?

'Majority at K' refers to getting at least one correct answer out of K samples, indicating consistency. 'Pass at K' signifies getting the correct answer on the first try (K=1), indicating fundamental capability improvement.

Can train time scaling techniques teach models to solve entirely new problems?

Currently, train time scaling methods primarily improve a model's ability to solve problems within its existing capabilities, making it more consistent or efficient. They do not yet fundamentally teach the model to solve entirely new problems or generalize significantly out-of-domain.

What are the key open problems in train time scaling research?

Open problems include understanding why 'pass at K' accuracy doesn't increase as much as 'majority at K', verifying if emergent reasoning behaviors are real, and developing better techniques for learning from model failures instead of just filtering them out.

When should one choose Supervised Fine-Tuning (SFT) over Reinforcement Learning (RL)?

SFT is often faster and more efficient if abundant high-quality data is available, but it may not boost reasoning capabilities. RL is more data-efficient for improving reasoning in domains with strong reward signals, but requires more careful implementation and tuning.

Key Moments

Stanford CS329A Self-Improving AI Agents | Part 6 | Train Time Scaling/Scaling RL

Stanford Online

Education5 min read73 min video

Aug 3, 2026|183 views|16

Stanford Stanford Online

Save to Pod

Want to know something specific about what's covered?

We've already dissected every moment. Ask and we will deliver (with timestamps).

Key Moments

TL;DR

Train-time scaling allows smaller AI models to outperform larger ones on reasoning tasks by learning from their own outputs, but success hinges on robust verification and careful RL implementation.

Key Insights

DeepSeek Math, a 7B parameter model, achieved 51.7% accuracy on the Amy benchmark using train-time scaling, significantly outperforming a 175B parameter GPT-3.5 model's estimated 5% accuracy.

The STAR paper demonstrates that fine-tuning a model on self-generated reasoning chains, even for failed attempts (by explaining backwards from the correct answer), can bootstrap reasoning capabilities and improve sample efficiency.

DeepSeek Math's success involved starting with a code-pre-trained model (DeepSeek Coder) and curating extensive web math data, showing that code-to-math transfer and data quality are crucial priming steps before RL.

GRPO (Generalized Proximal Policy Optimization) reduces memory requirements for RL by using a generalized advantage estimation instead of a separate critic, allowing for more scalable reinforcement learning for language models.

The DAPO paper improves RL stability for harder reasoning problems by introducing asymmetric clipping for exploration, dynamic sampling to maintain effective batch size, and token-level loss to control response length and prevent gradient collapse.

While train-time scaling, especially RL, can significantly boost performance on verifiable domains like math and coding, it requires careful implementation, robust reward signals, and is more difficult to get right than supervised fine-tuning for general capabilities.

Train-time scaling enables smaller models to achieve high reasoning accuracy

The lecture introduces train-time scaling as a method to significantly improve the reasoning capabilities of AI models, particularly smaller ones, by having them learn from their own outputs. This contrasts with traditional scaling based on model parameters. For instance, the Amy benchmark, designed for complex mathematical reasoning, shows dramatic performance gains: a 7B parameter DeepSeek Math model achieves over 51.7% accuracy, far surpassing the estimated 5% accuracy of a 175B parameter GPT-3.5. This highlights that compute invested in iterative refinement and learning from self-generated data can be more impactful than simply increasing model size. The core idea is to close the loop, using filtered model outputs to further train the model, a process that can substitute for model parameters and is particularly effective in domains with verifiability, such as mathematics and coding.

STAR: Bootstrapping reasoning with self-generated rationales

The STAR paper offers a simple yet effective method for enhancing a model's reasoning abilities by generating and learning from its own reasoning chains. Unlike relying solely on internet-scale data (which often lacks explicit reasoning steps) or expensive manual annotation, STAR starts with a small set of reasoning examples. It generates solutions, filters for correctness, and then, crucially, for incorrect attempts, it uses the correct answer as a hint to generate rationales backwards. This "explain-backwards" approach bootstraps the model's reasoning capabilities, expanding its training data to include more difficult problems. The assumption is that if the final answer is correct, the reasoning path is likely of good quality, although this proxy has limitations. This iterative process allows models to progressively improve their reasoning, demonstrating high sample efficiency by leveraging self-generated, high-quality training data.

DeepSeek Math: Priming for RL with code transfer and curated data

DeepSeek Math illustrates how to effectively prepare a model for reinforcement learning on mathematical reasoning tasks. Instead of training on academic papers (like arXiv), they found significant benefits from curating math-related content from common crawl web pages, starting with a code-pre-trained model (DeepSeek Coder). This "code-to-math" transfer proved more effective than typical academic training data, enhancing the model's ability to reason and use tools. By curating high-quality data with better coverage across mathematical domains, they primed the model before applying RL. This emphasizes that the model's foundational capabilities in related areas and the quality of its specialized training data are critical prerequisites for successful RL-based improvements in complex domains.

GRPO: Efficient reinforcement learning for language models

Reinforcement learning on large language models can be memory-intensive due to the need to maintain multiple model copies (policy, critic, reward model). GRPO (Generalized Proximal Policy Optimization) addresses this by proposing a new technique that uses a generalized advantage estimation (GAE) instead of a separate critic. This reduces the number of required models from four to three. The core idea is to normalize rewards by subtracting the mean and dividing by the standard deviation, creating an advantage function that allows for more memory-efficient scaling of RL. DeepSeek Math successfully employed GRPO to achieve a 51.7% accuracy on the math benchmark with a 7B model, demonstrating that this approach can be effective even with limited computational resources, making RL more accessible for training powerful language models on reasoning tasks.

DAPO: Stabilizing RL for complex reasoning chains

The DAPO paper tackles the instability and challenges encountered when scaling RL algorithms like GRPO to harder reasoning problems, often involving longer reasoning chains. Key innovations include asymmetric clipping in the policy gradient update to prevent exploration collapse and allow for larger improvements, and dynamic sampling to maintain an effective batch size by filtering out completely correct or incorrect samples. DAPO also introduces a token-level loss, instead of a sample-level loss, to control response length and entropy, preventing uncontrolled growth and mitigating noise from truncated reasoning chains. These techniques were applied to a 32B parameter model (Quen), pushing its performance on the Amy benchmark from 30% to 50%, significantly improving stability and allowing the model to effectively hill-climb on complex problems.

Comparing train-time scaling techniques and their applications

The lecture contrasts three main approaches to train-time scaling: STAR, GRPO (as used in DeepSeek Math), and DAPO. STAR is recommended for simpler reasoning tasks and when RL infrastructure is unavailable, offering good sample efficiency with limited data. GRPO is suitable for standard math reasoning when a strong base model is available and good instruction data is used for priming, providing a scalable RL solution even with memory constraints. DAPO is essential for state-of-the-art performance on competition-level problems (like Amy) requiring long and complex reasoning chains, where precise control over RL variables like entropy and response length is critical. While all techniques can improve majority-k performance and output coherence, they may not fundamentally improve out-of-domain generalization or teach the model entirely new problem-solving paradigms. RL, in particular, requires significant effort to implement correctly but offers the potential to improve performance with fewer examples when a strong reward signal exists, whereas SFT is faster if high-quality data is abundant.

Mentioned in This Episode

●Software & Apps

●Companies

●Studies Cited

●Concepts

Train Time Scaling Techniques for AI Reasoning

Practical takeaways from this episode

Do This

Leverage train time scaling to improve model reasoning capabilities, especially in domains with verifiability.

Consider using reasoning chains (like Chain of Thought) to boost model performance on complex problems.

For smaller datasets or limited RL infrastructure, the STAR method offers a way to bootstrap reasoning capabilities by generating rationales.

When using DeepSeek Math's GRPO, prime the model with good instruction data and start from a code-trained model for better math reasoning.

For harder problems requiring longer reasoning chains, employ advanced RL techniques like those in DAPO, focusing on asymmetric clipping and dynamic sampling.

Balance train time compute with test time compute to optimize overall model performance.

If using RL, ensure a strong reward signal and consider techniques to maintain stable entropy and control response length.

Avoid This

Do not solely rely on pre-training on internet data for advanced reasoning capabilities.

Avoid manually annotating reasoning steps if cost is a concern; explore automated or bootstrapped methods.

Do not assume standard RL algorithms (like PO) will scale effectively without memory optimizations for larger models.

Be cautious of collapsed entropy and uncontrollable response lengths when scaling RL algorithms naively.

Do not solely rely on the final answer's correctness as a proxy for reasoning quality; consider evaluating the reasoning chains themselves.

Avoid filtering out all 'zero reward' or 'one reward' samples in dynamic sampling, as they can still provide useful signals.

Common Questions

Test time scaling involves techniques like majority voting or inference-time sampling to improve model output quality after training. Train time scaling, conversely, uses the filtered outputs from test time scaling (or other methods) to further fine-tune and improve the model during its training phase, creating a self-improvement loop.

Topics

Reinforcement Learning AI & Machine Learning Programming & Software Science & Mathematics Language Models Model Training Reasoning Capabilities Self-improving AI Mathematical Reasoning Train Time Scaling

Mentioned in this video

Software & Apps

DeepSeek Math

A paper and model that focuses on mathematical reasoning and uses a novel reinforcement learning approach to improve performance.

GPT-4o

A model used as a baseline for comparison against thinking models, particularly in domains with verification.

AlphaCode

A system mentioned in the context of making search infinite for solutions at test time.

DAPO

The third paper discussed, focusing on reinforcement learning techniques to stabilize and improve reasoning with longer chains.

Quen 32B

A model that achieved 50% accuracy on the Amy benchmark when using the DAPO technique.

Grock

A model claimed to use a significant percentage of RL in its training, though performance gains were debated.

GPT-J

A 6 billion parameter open-source model based on GPT-3, used for experiments with the STAR approach.

Osiris

A reasoning model mentioned as an example of models utilizing reasoning capabilities.

MINERvA

A paper that improved the Palm model on STEM tasks by training on science and math data.

GRPO

A proposed technique in DeepSeek Math that reduces memory requirements in RL by using a generalized advantage estimation instead of a critic.

RLeF

Reinforcement Learning from Feedback, mentioned as a paper that uses execution feedback for rewards.

STAR

A paper focusing on boosting the reasoning capabilities of models by incorporating rationales into problems, used as the first paper discussed.

DPO

A typical reinforcement learning algorithm used in RLHF that involves keeping multiple policy models, critic, and reward models.

Gemini Flash Thinking

A reasoning model mentioned as an example of models utilizing reasoning capabilities.

GPT-3.5

A 175 billion parameter model that achieved approximately 5% accuracy on the Amy benchmark.

PaLM

A model that was the basis for Minerva's improvements on STEM tasks, and upon which DeepSeek Math built.

Grock 4

A specific version of Grock claimed to use 50% RL, but the performance jump was not as expected due to bottlenecks.

Companies

DeepSeek

A company whose 7B model, DeepSeek Math, achieved 51.7% accuracy on the Amy benchmark using train time scaling.

Studies & Research

GSM8K

A dataset of grade-school math problems used to test the STAR approach.

Common Sense QA

A dataset of common sense reasoning problems used to evaluate the STAR approach.

Concepts

Amy

A benchmark for mathematical reasoning problems, used to evaluate the performance of different models and techniques discussed in the lecture.

Chain of Thought

A technique where models are prompted to show their work step-by-step, which improves interpretability and mathematical reasoning.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free