I

Integer Tokenization is Insane (blog post)

Study / ResearchMentioned in 1 video

A blog post investigating how numbers are tokenized (variable tokenization for integers) and its impact on arithmetic performance.

Videos Mentioning Integer Tokenization is Insane (blog post)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

A blog post investigating how numbers are tokenized (variable tokenization for integers) and its impact on arithmetic performance.