AdamW

ConceptMentioned in 2 videos

Optimizer chosen for training (AdamW variant) with recommended betas and epsilon following GPT-3 guidance.

Videos Mentioning AdamW

Let's reproduce GPT-2 (124M)

Let's reproduce GPT-2 (124M)

Andrej Karpathy

Optimizer chosen for training (AdamW variant) with recommended betas and epsilon following GPT-3 guidance.

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

⚡ Open Model Pretraining Masterclass — Elie Bakouch, HuggingFace SmolLM 3, FineWeb, FinePDF

Latent Space