AdamW

Concept

Optimizer chosen for training (AdamW variant) with recommended betas and epsilon following GPT-3 guidance.

Mentioned in 2 videos

Videos Mentioning AdamW

Let's reproduce GPT-2 (124M)

Let's reproduce GPT-2 (124M)

Andrej Karpathy

Optimizer chosen for training (AdamW variant) with recommended betas and epsilon following GPT-3 guidance.

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

Latent Space