G

GPT-4 tokenizer whitespace improvements (design choice)

Tool / ProductMentioned in 1 video

Design choice in CL100k/GPT‑4 tokenizer to group repeated spaces (improves code density), explicitly shown with Python indentation examples.

Videos Mentioning GPT-4 tokenizer whitespace improvements (design choice)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Design choice in CL100k/GPT‑4 tokenizer to group repeated spaces (improves code density), explicitly shown with Python indentation examples.