t

tiktoken library

Tool / ProductMentioned in 1 video

Official OpenAI tokenization library (in Rust) for inference with GPT tokenizers; discussed for usage and special-token handling.

Videos Mentioning tiktoken library

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Official OpenAI tokenization library (in Rust) for inference with GPT tokenizers; discussed for usage and special-token handling.