FineWeb / fineweb-edu

Concept

High-quality filtered subset of Common Crawl (fineweb-edu) used in the video as a 10B-token sample for pretraining experiments.

Mentioned in 1 video

Videos Mentioning FineWeb / fineweb-edu

Let's reproduce GPT-2 (124M)

Let's reproduce GPT-2 (124M)

Andrej Karpathy

High-quality filtered subset of Common Crawl (fineweb-edu) used in the video as a 10B-token sample for pretraining experiments.