S

SentencePiece

Tool / ProductMentioned in 1 video

Google's library used for both training and inference (used by LLaMA and others); video explains its different design (codepoint-level BPE, byte-fallback, many options).

Videos Mentioning SentencePiece

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Google's library used for both training and inference (used by LLaMA and others); video explains its different design (codepoint-level BPE, byte-fallback, many options).