tiktoken

Let's reproduce GPT-2 (124M)

Andrej Karpathy

OpenAI's tokenizer library used to encode text into GPT-2 token IDs for sampling and training.