numerical stability trick: subtract max(logits)

Tool / ProductMentioned in 1 video

Subtracting per-row max from logits before exponentiation to avoid overflow; its backward contribution is small/near-zero and discussed in detail.

Videos Mentioning numerical stability trick: subtract max(logits)

Building makemore Part 4: Becoming a Backprop Ninja

Building makemore Part 4: Becoming a Backprop Ninja

Andrej Karpathy

Subtracting per-row max from logits before exponentiation to avoid overflow; its backward contribution is small/near-zero and discussed in detail.