tiktoken library

Software / App

Official OpenAI tokenization library (in Rust) for inference with GPT tokenizers; discussed for usage and special-token handling.

Mentioned in 1 video

Videos Mentioning tiktoken library

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Official OpenAI tokenization library (in Rust) for inference with GPT tokenizers; discussed for usage and special-token handling.