SentencePiece

Software / App

Google's library used for both training and inference (used by LLaMA and others); video explains its different design (codepoint-level BPE, byte-fallback, many options).

Mentioned in 1 video

Videos Mentioning SentencePiece

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Google's library used for both training and inference (used by LLaMA and others); video explains its different design (codepoint-level BPE, byte-fallback, many options).