t

tiktoken (web app: tiktokenizer/bell.app)

Tool / ProductMentioned in 1 video

OpenAI's tokenization tooling and a live web demo used in the video to visualize tokenization (GPT-2 vs CL100K/GPT‑4 tokenizers).

Videos Mentioning tiktoken (web app: tiktokenizer/bell.app)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

OpenAI's tokenization tooling and a live web demo used in the video to visualize tokenization (GPT-2 vs CL100K/GPT‑4 tokenizers).