Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 15: Mid/Post-Training

A large language model that represents a strong base model, but with limited utility and difficulty in instruction following compared to newer models. Its primary uses were copywriting and simple tasks.

A successor to GPT-3 that offered a significant improvement in interaction and instruction following, making it seem amazing to early users. It is a benchmark for current language model capabilities.

Mentioned as a significant advancement that enabled better instruction following with long, programmatic prompts, contrasting with the limitations of earlier models like GPT-3.

A model exhibiting strong instruction following capabilities, capable of 'oneshotting' complex prompts. It serves as a benchmark for which other models are compared against, even in reverse engineering efforts.

A language model trained by Google using the FLAN dataset. The FLAN dataset's structure, derived from existing benchmarks, led to some unnatural task formulations.

Founded by Berkeley researchers, Alpaca used distillation from ChatGPT traces to create input-output pairs, demonstrating that such chat-style data could effectively train ChatGPT-like systems when applied to models like LLaMA.

A language model from Berkeley that utilized online user-shared prompts as inputs for distillation.

A large-scale, crowdsourced effort to build a high-quality, human-driven instruction tuning dataset, akin to Wikipedia in its collaborative model, aiming to match the performance of closed-source labs.

A newer generation of instruction tuning datasets that employs increasingly sophisticated methods for generating instruction-following data using language models.

Presented as a reference for a performant post-training pipeline, including a safety component with approximately 50,000 examples and utilizing a 'wild chat' dataset for mining unsafe behaviors.

NVIDIA's open-source initiative that incorporates a substantial amount of agentic SFT data, including tool calls alongside textual responses.

Used as an example to illustrate how different models exhibit distinct tones, highlighting conscious decisions made in data collection regarding chatbot style.

Its description of safety SFT is presented as one of the more detailed publicly available explanations, though it lacks specifics on the number of examples used.

A previous project that provided free chat access to users, collecting interactions to filter out unsafe behaviors and jailbreaks, which were then used to generate SFT data for Tulu 3's safety component.

Its appendix provides a glimpse into industry data collection processes for RLHF, detailing guidelines for rating outputs on helpfulness, truthfulness, and harmlessness.

Annotation data for Bard was leaked, showing a similar structure to InstructGPT's guidelines, focusing on helpfulness and presentation using a Likert scale rather than pairwise feedback.

Trust Region Policy Optimization, an off-policy RL algorithm that takes multiple steps while staying close to the original policy using importance weighting corrections.

Proximal Policy Optimization, a reinforcement learning algorithm that aims to improve stability by using a clipping heuristic to discourage large policy changes. It's a key algorithm in RLHF.

An open-source model Hugging Face attempted to build without model distillation, prioritizing human data collection, but ultimately found human data to be less efficient than AI feedback.

Direct Preference Optimization, a simpler RLHF algorithm that eliminates the need for a separate reward model and on-policy sampling. It works by taking gradient steps towards preferred responses and negative steps away from dispreferred ones.

Models like LLaMA were trained using DPO for their core RLHF primitive, demonstrating its effectiveness. LLaMA 7B is mentioned as a respectable open-source model size.

A dataset generation approach where a model is used to generate its own high-quality responses to inputs, based on the idea that models themselves might surpass human annotators.

An early work by Anthropic that involved prompting a model to generate safety data, which was then used to train the model itself, creating a self-post-training data generation loop for safer outputs.

Mentioned for its open-source efforts, specifically Neotron, which includes a significant portion of agentic SFT examples featuring tool calls.

Researchers at Meta conducted ablations and used court documents from a lawsuit regarding the use of books in training data to estimate the usefulness of different book subsets for model training.

Mentioned for its work in self-verification using models, particularly in domains like mathematics where verifying a proof is easier than generating one.

Mentioned for its development of InstructGPT, its pioneering work in RLHF, and its models like GPT-4. They also noted that their RLHF models were uncalibrated.

Attempted to build an open-source model called Zephier without model distillation, relying solely on human data collection. They found human data collection to be time-consuming, costly, and not superior to model-based annotations, eventually using AI feedback.

Pioneered self-training with constitutional AI, prompting models to generate safety data. They also noted models can be naturally uncalibrated after RLHF.

Its internal documents were leaked, revealing frank discussions about post-training data, particularly in efforts to improve Google Bard. The company was also involved in low-cost annotation outsourcing.

A proponent of reinforcement learning who argues that RL and training are necessary to teach models what they know and don't know, and to calibrate them.

A dataset used as a source for the FLAN dataset, where email bodies and subjects were repurposed into a supervised task for generating subject lines.

A large dataset used for summarization tasks, potentially a source for the summarization examples within the FLAN dataset.