tiktoken (tokenizer demo)

Deep Dive into LLMs like ChatGPT

Andrej Karpathy

Website/tool used to inspect tokenization (demonstrated with GPT-style tokenizer).