e

encoder.json & vocab.bpe (OpenAI tokenizer artifacts)

Tool / ProductMentioned in 1 video

Files used by OpenAI to store the trained tokenizer: encoder (ID→string) and vocab.bpe (merge list). Described as the two items needed to represent a trained tokenizer.

Videos Mentioning encoder.json & vocab.bpe (OpenAI tokenizer artifacts)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Files used by OpenAI to store the trained tokenizer: encoder (ID→string) and vocab.bpe (merge list). Described as the two items needed to represent a trained tokenizer.