What is zero-shot learning and few-shot learning in LLMs?

Zero-shot learning is when a model can perform a task without any prior examples, just based on the task description. Few-shot learning involves providing the model with a few examples of the task to help it understand and perform better.

What is chain of thought reasoning in LLMs?

Chain of thought reasoning involves guiding an LLM through a step-by-step process to arrive at an answer, similar to how humans might break down a problem. This emergent behavior in larger models significantly improves their reasoning capabilities, especially for complex tasks.

What is the difference between pre-training and fine-tuning for LLMs?

Pre-training is the initial phase where models learn general language understanding by predicting the next token on vast amounts of text. Fine-tuning then adapts the model for specific tasks or preferences, using higher-quality data, instruction tuning, or RLHF.

What is Reinforcement Learning from Human Feedback (RLHF)?

RLHF is a fine-tuning process where human preferences are used to create a reward model. This reward model then guides the LLM to generate outputs that are more aligned with human values, such as correctness, helpfulness, or harmlessness.

What is inference scaling?

Inference scaling involves improving model performance at inference time without changing model parameters. Techniques like parallel sampling or repeated generation allow models to explore more possibilities and achieve better results than a single pass.

How do agentic workflows differ from standard LLM chatbots?

Agentic workflows allow LLMs to accomplish tasks end-to-end by planning steps, interacting with environments, receiving feedback, and correcting actions. Unlike chatbots, which are often single-turn, agents are designed for goal-oriented task completion.

What are the key components of agentic workflows?

Agentic workflows often include LLM calls, verifiers (like unit tests), critics or judges (LLMs evaluating outputs), tool calls (e.g., web search), and orchestration mechanisms like prompt chaining or routing.

How are reasoning models trained?

Reasoning models are developed through a combination of techniques, including pre-training, instruction tuning with examples of reasoning (like chain of thought), and RLHF or RL from synthetic data. They learn skills like planning, task decomposition, and self-correction.

What are the course requirements for CS 329A?

The course involves three homework assignments and a course project. The project allows students to explore agentic systems or delve deeper into research questions, with opportunities for collaboration and API credits.

What kind of projects are acceptable for CS 329A?

Acceptable projects include developing new evaluation datasets, assessing the reliability of existing agentic systems, improving benchmarks, or critically evaluating and extending research papers covered in the course. Projects should be research-oriented, not just simple app implementations.

Key Moments

Stanford CS329A Self-Improving AI Agents | Part 1 | Course Overview

Q: How did ChatGPT achieve its success beyond just scaling?

ChatGPT's success was driven by innovations beyond model scaling, specifically instruction tuning and reinforcement learning from human feedback (RLHF). These techniques align the model's outputs with human preferences and instructions.

Stanford Online

Education5 min read70 min video

Aug 3, 2026|1,235 views|98|5

Stanford Stanford Online

Save to Pod

Key Moments

TL;DR

Self-improving AI agents are emerging, capable of complex real-world workflows and sophisticated reasoning, but their development and verification remain significant research challenges.

Key Insights

Increasing model size (parameters), compute, and data has historically driven performance improvements in LLMs, with exponential growth in parameters from BERT (340M) to GPT-4 (trillions).

Emergent behaviors like chain-of-thought reasoning appear in larger models, enabling them to break down problems and show their work, significantly improving performance on complex tasks.

ChatGPT's rapid adoption (1M users in 5 days) was attributed to innovations beyond scaling, specifically instruction tuning and Reinforcement Learning from Human Feedback (RLHF).

Inference scaling, through methods like repeated sampling, can unlock more capability from existing models without changing parameters, demonstrating that models often know more than initially apparent.

Agentic workflows, moving beyond chatbots, enable end-to-end task completion by planning, interacting with environments, and self-correcting based on feedback, with applications in coding, research, and customer support.

Verifiers and feedback loops are crucial for improving AI agents, especially in domains like coding where unit tests can provide robust validation, though human feedback remains a bottleneck for creative tasks.

The scaling laws that powered large language models

The first lecture of Stanford's CS329A course introduces the foundational trend in large language models (LLMs): the power of scaling. For years, increasing the number of parameters, the amount of compute, and the size of the training dataset consistently led to better performance and lower loss functions. This relationship, often referred to as scaling laws, has been the driving force behind models like BERT, GPT-3, PaLM, and Gemini. For instance, model sizes have grown exponentially, from BERT's 340 million parameters to GPT-3's 175 billion, and estimates for GPT-4 suggesting trillions. This exponential growth in scale has been directly correlated with improved performance on natural language benchmarks, reasoning tasks, and the emergence of few-shot and zero-shot learning capabilities, where models can perform tasks with minimal or no specific training examples.

Emergent behaviors: Few-shot learning and chain-of-thought reasoning

Beyond predictable performance gains, scaling LLMs has led to surprising emergent behaviors. One significant capability is few-shot learning, where models can adapt to new tasks by seeing just a few examples within the prompt, drastically reducing the need for extensive fine-tuning. Even more remarkable is the emergence of chain-of-thought (CoT) reasoning. While smaller models struggle with multi-step problems, larger models, when prompted with examples that include the reasoning process (e.g., step-by-step problem-solving), can learn to generate similar reasoning chains themselves. This ability to articulate a thought process, demonstrated on tasks like math word problems, has been crucial for improving the accuracy and reliability of LLMs on complex reasoning tasks. The lecture highlighted that capabilities like CoT do not linearly improve with scale but rather appear at certain model sizes, making their discovery a testament to empirical exploration rather than pure prediction.

ChatGPT's innovations: Instruction tuning and RLHF

The rapid success of ChatGPT, which reached 1 million users in just 5 days, was not solely due to scaling but also key innovations in the fine-tuning process. The lecture detailed two critical steps: instruction tuning and Reinforcement Learning from Human Feedback (RLHF). Instruction tuning teaches models to follow explicit instructions and respond in a desired format, often using a mix of human-generated and synthetic data containing instruction-answer pairs. RLHF goes further by using human preferences to train a reward model, which then guides the LLM to generate responses that are more aligned with human values like helpfulness, harmlessness, and sensibility. This alignment process is essential for steering powerful, pre-trained models towards safe and useful interactions, though it remains an ongoing challenge.

Inference scaling: Unlocking latent capabilities

A more recent frontier is inference scaling, which focuses on improving model performance at inference time without altering the model's parameters. The 'Large Language Monkeys' project, inspired by the infinite monkey theorem, demonstrated that by asking an LLM to solve a problem multiple times (parallel sampling) and then selecting the best response using a verifier (like unit tests for code), significantly better results can be achieved. For instance, smaller models like LLaMA 8B, when sampled 10,000 times, outperformed GPT-4 with a single sample on certain benchmarks. This suggests that models possess more latent capabilities than a single forward pass reveals, and techniques like inference scaling can extract this hidden potential. This approach is distinct from traditional training and opens new avenues for enhancing AI performance.

The rise of agentic workflows

The course transitions to the concept of self-improving AI agents, moving beyond simple chatbots to systems capable of executing complex, end-to-end real-world workflows. Agents are defined by their ability to plan, take actions, interact with an environment, receive feedback, and self-correct to achieve a given goal. Examples like Claude Code and Deep Research illustrate this shift, enabling tasks like automated research or complex coding modifications solely through English instructions. This paradigm shift requires models to possess planning capabilities, multi-step reasoning, and robust self-improvement mechanisms, often involving interactions with external tools and maintaining memory of their ongoing tasks.

Orchestration and verification in agentic systems

Agentic workflows are orchestrated through various methods, including prompt chaining, routing, parallelization, and dedicated orchestrator models. A key component is verification: ensuring the agent's outputs are correct and reliable. In domains like coding, unit tests serve as powerful verifiers, allowing agents to generate code and then test it. For more creative or open-ended tasks, however, verification becomes a bottleneck, often relying on human feedback or LLM-based judges. The challenge of the 'generator-verifier gap' highlights the need for robust feedback loops. The course will explore how these systems are built, emphasizing the importance of planning, multi-step reasoning, and self-correction, with dedicated lectures on topics like verifiers and RLHF.

Course structure and project opportunities

Stanford's CS329A course offers a comprehensive curriculum on self-improving AI agents, featuring lectures, guest speakers from AI labs, and hands-on projects. The course structure includes three homework assignments and a significant course project, encouraging students to explore agentic systems, develop new benchmarks, or improve existing methods. Students will have access to API credits and can work in teams. The project timeline includes proposal submission, midterm presentations, and a final poster session. The grading rubric allocates 50% to homeworks and 50% to the project, emphasizing research-oriented work rather than simple application development. The course aims to equip students with the knowledge to build and understand the next generation of intelligent agents.

Mentioned in This Episode

●Software & Apps

●Companies

●Organizations

●Studies Cited

●People Referenced

Common Questions

The primary scaling trends that lead to better language model performance are increasing the number of parameters, expanding the dataset size, and increasing the amount of compute used for training. These factors contribute to lower test loss and improved capabilities.

Topics

Ai Agents AI & Machine Learning Technology & Innovation Large Language Models Agentic Workflows Reasoning Models Chain-of-thought LLM Scaling Instruction Tuning Inference Scaling

Mentioned in this video

Software & Apps

CS 329A

The course on self-improving AI agents being taught.

Gemini

A model Azalia Mirhoseini worked on at Google DeepMind.

cs329a.stanford.edu

The website for the CS 329A course, containing lecture materials, papers, and schedule.

BERT

An older language model mentioned in the context of scaling parameters.

GPT-3

A foundational large language model that demonstrated performance improvements with scaling.

PaLM

A large language model mentioned in the context of scaling laws and emergent behaviors.

ChatGPT

A highly successful AI chatbot launched in November 2022, known for its rapid user acquisition and innovations like instruction tuning and RLHF.

GPT-4

An estimated large language model with trillions of parameters, discussed in the context of exponential growth in model size.

GPT-4o

A model that reasoning models are compared against, showing outperformance in reasoning tasks but not necessarily in creative writing.

Canvas

The platform used for course updates, assignments, and office hours.

Gradescope

The platform for submitting project milestones and homeworks.

Llama 8B

A language model whose performance on benchmarks can improve significantly with increased sampling during inference.

Gemini Thinking

A model series that integrates fine-tuning and test-time scaling, contributing to advancements in reasoning models.

LLaMA 3 8B

A language model whose performance on benchmarks can improve significantly with increased sampling during inference.

Claude Code

A coding agent that enables real-world workflows by planning, interacting with the environment, and achieving tasks end-to-end.

Codex

An OpenAI coding agent that allows users to modify files and figure out test cases using English instructions.

Code Monkeys

A project from the previous year that explored generating unit tests for model-generated code to aid verification.

EdStem

A platform for asking and broadcasting course-related questions to the community.

Organizations

Google Brain

Where instructors Akanksha and Azalia Mirhoseini met.

Google DeepMind

Where instructors Akanksha and Azalia Mirhoseini worked and where Azalia worked on Gemini.

Companies

OpenAI

Developer of Codex, a coding agent, and GPT-4o.

Reflection AI

A startup where one of the instructors, Akanksha, is in research.

Anthropic

Mentioned as a place where Azalia Mirhoseini worked.

DeepSeek

A model mentioned as bringing together fine-tuning and test-time scaling for improved reasoning and synthetic data generation.

People

Azalia Mirhoseini

Assistant professor in the CS department at Stanford and co-instructor for CS 329A.

William Shakespeare

Mentioned in the context of the infinite monkey theorem, illustrating the idea that random typing can eventually produce great works.

Studies & Research

GSM8K

A dataset that first showed signs of chain-of-thought reasoning capabilities in large language models.

Ask anything from this episode.

Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.

Get Started Free