Integer Tokenization is Insane (blog post)

Study / Research

A blog post investigating how numbers are tokenized (variable tokenization for integers) and its impact on arithmetic performance.

Mentioned in 1 video

Videos Mentioning Integer Tokenization is Insane (blog post)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

A blog post investigating how numbers are tokenized (variable tokenization for integers) and its impact on arithmetic performance.