QK norm

Concept

A normalization step in Qwen 3 that dynamically rescales query and key vectors to maintain constant magnitudes, replacing QKV bias.

Mentioned in 1 video

Videos Mentioning QK norm

OpenAI vs. Deepseek vs. Qwen: Comparing Open Source LLM Architectures

OpenAI vs. Deepseek vs. Qwen: Comparing Open Source LLM Architectures

Y Combinator

A normalization step in Qwen 3 that dynamically rescales query and key vectors to maintain constant magnitudes, replacing QKV bias.