Rotary Positional Embeddings

Concept

Embeddings used in GPT OSS that encode token position directly into the attention mechanism to support longer contexts.

Mentioned in 1 video

Videos Mentioning Rotary Positional Embeddings

OpenAI vs. Deepseek vs. Qwen: Comparing Open Source LLM Architectures

OpenAI vs. Deepseek vs. Qwen: Comparing Open Source LLM Architectures

Y Combinator

Embeddings used in GPT OSS that encode token position directly into the attention mechanism to support longer contexts.