Key Moments
Stanford CME 295 Transformers & LLMs | Fall 2026 | Lecture 3 - LLM training
Stanford OnlineKey Moments
Training large language models involves pretraining, midtraining, and fine-tuning stages, with an increasing focus on reasoning and specialized instruction following. However, ensuring data quality and preventing model collapse remain significant challenges.
Key Insights
Training an LLM requires a multi-stage process: pretraining on vast text and code, midtraining to inject specific knowledge and longer contexts, and fine-tuning for specific tasks or behaviors.
The Chinchilla paper suggests a ratio of approximately 20 tokens per parameter for optimal training, indicating many scaled-up architectures might be undertrained.
Cross-entropy loss is used during pretraining to maximize the probability of the model seeing the training data, by predicting the next token in a sequence.
Low-Rank Adaptation (LoRA) significantly reduces the number of parameters that need to be trained during fine-tuning by decomposing weight matrices into lower-rank components.
Reinforcement Learning from Human Feedback (RLHF) involves training a reward model and then using reinforcement learning to align the LLM with human preferences, a complex process that can be approximated by Direct Preference Optimization (DPO).
Models are increasingly trained for 'reasoning' capabilities, enabling them to solve problems that require sequential thought processes, evidenced by techniques like Chain-of-Thought prompting and 'thinking tokens'.
The multi-stage process of LLM training
The training of large language models (LLMs) is a complex, multi-stage process. Traditionally, models were trained for specific use cases, but the current paradigm emphasizes a large, one-time pretraining phase to learn language broadly. This learned knowledge is then transferred to various downstream tasks. The lecture outlines three main stages: pretraining, midtraining, and fine-tuning (SFT), along with preference tuning. Pretraining focuses on language acquisition, midtraining injects domain-specific knowledge and longer context capabilities, and fine-tuning adapts the model to desired behaviors and tasks, such as acting as a helpful assistant. The ultimate goal is to create models that not only possess vast knowledge but can also reason and follow instructions effectively.
Pretraining: Learning the language
The initial stage, pretraining, involves exposing the LLM to massive amounts of text and code from the internet and other sources. The primary objective is to learn language patterns by predicting the next token in a sequence. This is often referred to as 'teacher forcing' during training, where the model always receives the correct previous token as input for the next prediction, ensuring it learns to predict the actual next token in the text. The loss function used is typically cross-entropy, which aims to maximize the probability of the model observing the training data. This process is highly parallelizable, allowing for efficient training on large datasets. The scale of data is immense, often in the trillions of tokens, and the Chinchilla paper suggests a ratio of about 20 tokens per parameter for optimal training, highlighting that many large models might be undertrained.
Midtraining: Enhancing knowledge and context
Midtraining is a more recent addition to the LLM training pipeline, aimed at injecting specific knowledge and improving the model's ability to handle longer contexts. This stage utilizes higher-quality, more task-aligned data. For instance, if a model needs to excel at mathematical reasoning, it would be exposed to more mathematical texts and code. A significant challenge addressed in this phase is long-context training. While pretraining often uses sequences of a few thousand tokens for efficiency, real-world applications may require processing tens or hundreds of thousands of tokens. Techniques like RoPE (Rotary Positional Embeddings) are adapted, often through frequency interpolation (e.g., 'Yarn'), to help models better learn positional differences in longer contexts without suffering from 'context collapse' or forgetting distant information. This stage also involves careful data mixture selection, experimenting with different data proportions to find optimal configurations for training.
Supervised fine-tuning for desired behaviors
Supervised Fine-Tuning (SFT) shifts the focus from broad language understanding to aligning the model's behavior with specific desired outputs, such as acting as a helpful assistant. This involves creating pairs of instructions (prompts) and desired responses. The model is then trained to predict the next token of the response given the instruction. While the underlying loss function often remains cross-entropy, the input data is structured as instruction-response pairs, rather than just contiguous text. The scale of data for SFT is significantly smaller than for pretraining, ranging from thousands to millions of examples, depending on the task's complexity and generality. A key challenge is ensuring the quality and diversity of this data, as the model will closely mimic its supervised training. Low-Rank Adaptation (LoRA) is a popular technique used during SFT to make the process more computationally efficient by only updating a small subset of the model's parameters.
Preference tuning: Aligning with human values
Preference tuning, notably through Reinforcement Learning from Human Feedback (RLHF) and its variants like Direct Preference Optimization (DPO), aims to align LLMs with human preferences and values. This is crucial because simply predicting the next token or following instructions doesn't guarantee helpful, harmless, or honest outputs. The process typically involves collecting pairwise preference data, where humans rate which of two model responses is better for a given prompt. A reward model is trained on this data, and then reinforcement learning is used to update the LLM to maximize this reward. This stage addresses issues where a model might produce undesirable outputs even after SFT, by explicitly teaching it what not to do. The complexity of RLHF has led to methods like DPO, which achieve similar alignment goals through a supervised learning approach, simplifying the training process.
The rise of reasoning capabilities
A significant trend in LLM development is the enhancement of reasoning abilities, moving beyond simple knowledge recall to problem-solving. Techniques like Chain-of-Thought (CoT) prompting encourage models to output intermediate reasoning steps before providing a final answer. Initially, this was achieved through few-shot prompting, but newer models are trained to generate these reasoning chains intrinsically. This is particularly important for tasks like mathematics and coding, where sequential logic is essential. The training often involves using verifiable rewards, where the correctness of the final answer and the presence of a reasoning trace can be objectively assessed. Methods like GRPO (Grouped Policy Optimization) are employed, using reinforcement learning to maximize advantages while staying close to a reference model, ensuring improved reasoning without deviating too much from the model's learned knowledge.
Challenges in data and training
Several challenges persist in LLM training. Data quality is paramount; contaminated or low-quality data can lead to 'model collapse,' where the model's outputs degrade over time, failing to generalize. Ensuring diverse and high-quality datasets requires significant effort in filtering and curation. Furthermore, the computational cost of training these massive models is substantial, driving innovations like LoRA for parameter-efficient fine-tuning. The 'knowledge cutoff' date in trained models means their information is static, requiring methods for continuous learning or retrieval augmentation. Finally, understanding the 'secret sauce' of data mixtures and training strategies, often proprietary for closed-source models, remains an area of active research and development.
Mentioned in This Episode
●Software & Apps
●Companies
●Books
●Concepts
Common Questions
يتضمن تدريب النماذج اللغوية الكبيرة عادةً ثلاث مراحل رئيسية: التدريب المسبق لتعلم اللغة، والتدريب المتوسط لإدخال المعرفة الخاصة بالمهام، والضبط الدقيق (SFT) لتعديل سلوك النموذج ليناسب المهام المحددة.
Topics
Mentioned in this video
تقنية لتمديد ترددات RoPE لمساعدة النماذج على التعامل مع سياقات أطول بشكل أفضل.
تقنية لتضمينات الموضع في Transformer، لا تحتوي على معلمات متعلمة وتساعد النموذج على التعامل مع سياقات أطول.
بنية معمارية تُستخدم في النماذج اللغوية الكبيرة، تتكون من مشفر ومفكك ترميز. تم تحسينها بعدة طرق منذ إطلاقها الأول.
فئة من النماذج تختار خبراء محددين لتفعيلهم في تمريرة أمامية معينة لتحسين الأداء.
دالة خسارة تُستخدم لتعظيم احتمالية رؤية النموذج لبيانات التدريب، وتعتبر مقياسًا شائعًا في تدريب النماذج اللغوية.
وسيلة لتوفير المفاتيح والقيم اللازمة لتشفير التسلسل.
تقنية تكييف منخفض الرتبة تسمح بتعديل عدد أقل بكثير من المعلمات في النماذج اللغوية الكبيرة لتحسين الكفاءة والأداء.
صيغة تُستخدم في نمذجة المكافأة لتصميم احتمالية تفضيل شيء على شيء آخر.
نموذج مفتوح الأوزان يعرض خط أنابيب التدريب وإعداد البيانات المعقد.
نموذج مفتوح الأوزان حديث نسبياً تم تدريبه على 30 تريليون رمز، ويُستخدم كمثال لدراسة حالة في المحاضرة.
نموذج لغوي كبير استخدمت بيانات تدريبه حتى 30 أبريل 2026، مما يوضح أن معرفته ثابتة في الزمن.
نموذج من OpenAI كان يميز نموذج التفكير الخاص به بوضوح على واجهة المستخدم.
مصدر بيانات ويب ضخم يستخدم عادة في التدريب الأولي للنماذج اللغوية الكبيرة.
شركة ذكرت مقالة رائعة حول LoRA، ينصح بالاطلاع عليها.
شركة رائدة في مجال الذكاء الاصطناعي، كانت أول من تحدث عن تحسين الأداء في مهام الرياضيات والبرمجة بتقنيات التفكير، وأصدرت مؤخرًا حلولًا لمسائل معلقة.
أول نموذج مفتوح الأوزان في يناير 2025 يشرح آلية عمل نماذج التفكير.
منصة تستخدم كمصدر لبيانات البرمجة في التدريب الأولي للنماذج اللغوية الكبيرة.
More from Stanford Online
View all 150 summaries
79 minStanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Camera Models & Calibration
76 minStanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Overview, Mobile Robot Kinematics
75 minStanford AA274A Principles of Robotic Autonomy | Autumn 2019 | Trajectory Optimization
104 minStanford CME295 Transformers & LLMs | Autumn 2026 | Lecture 2 - Large Language Models
Ask anything from this episode.
Save it, chat with it, and connect it to Claude or ChatGPT. Get cited answers from the actual content — and build your own knowledge base of every podcast and video you care about.
Get Started Free