encoder.json & vocab.bpe (OpenAI tokenizer artifacts)

Software / App

Files used by OpenAI to store the trained tokenizer: encoder (ID→string) and vocab.bpe (merge list). Described as the two items needed to represent a trained tokenizer.

Mentioned in 1 video

Videos Mentioning encoder.json & vocab.bpe (OpenAI tokenizer artifacts)

Let's build the GPT Tokenizer

Let's build the GPT Tokenizer

Andrej Karpathy

Files used by OpenAI to store the trained tokenizer: encoder (ID→string) and vocab.bpe (merge list). Described as the two items needed to represent a trained tokenizer.