tiktoken

Software / App

OpenAI's tokenizer library used to encode text into GPT-2 token IDs for sampling and training.

Mentioned in 1 video

Videos Mentioning tiktoken

Let's reproduce GPT-2 (124M)

Let's reproduce GPT-2 (124M)

Andrej Karpathy

OpenAI's tokenizer library used to encode text into GPT-2 token IDs for sampling and training.