Delving Deep into Rectifiers

Study / Research

Paper (He et al.) analyzed initialization for ReLU-like nonlinearities and derived recommended gains (e.g., sqrt(2)) to preserve activation variance.

Mentioned in 1 video

Videos Mentioning Delving Deep into Rectifiers

Building makemore Part 3: Activations & Gradients, BatchNorm

Building makemore Part 3: Activations & Gradients, BatchNorm

Andrej Karpathy

Paper (He et al.) analyzed initialization for ReLU-like nonlinearities and derived recommended gains (e.g., sqrt(2)) to preserve activation variance.