D

Delving Deep into Rectifiers

Study / ResearchMentioned in 1 video

Paper (He et al.) analyzed initialization for ReLU-like nonlinearities and derived recommended gains (e.g., sqrt(2)) to preserve activation variance.

Videos Mentioning Delving Deep into Rectifiers

Building makemore Part 3: Activations & Gradients, BatchNorm

Building makemore Part 3: Activations & Gradients, BatchNorm

Andrej Karpathy

Paper (He et al.) analyzed initialization for ReLU-like nonlinearities and derived recommended gains (e.g., sqrt(2)) to preserve activation variance.