GPT-4 tokenizer whitespace improvements (design choice)

Software / App

Design choice in CL100k/GPT‑4 tokenizer to group repeated spaces (improves code density), explicitly shown with Python indentation examples.

Mentioned in 1 video

Videos Mentioning GPT-4 tokenizer whitespace improvements (design choice)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Design choice in CL100k/GPT‑4 tokenizer to group repeated spaces (improves code density), explicitly shown with Python indentation examples.