tiktoken (web app: tiktokenizer/bell.app)

Software / App

OpenAI's tokenization tooling and a live web demo used in the video to visualize tokenization (GPT-2 vs CL100K/GPT‑4 tokenizers).

Mentioned in 1 video

Videos Mentioning tiktoken (web app: tiktokenizer/bell.app)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

OpenAI's tokenization tooling and a live web demo used in the video to visualize tokenization (GPT-2 vs CL100K/GPT‑4 tokenizers).