Glorot

Glorot is a surname that has become notable in the field of machine learning through the work of Xavier Glorot. He is best known for co-authoring a 2010 paper with Yoshua Bengio that introduced a weight initialization scheme now commonly referred to as Glorot initialization or Xavier initialization. The method addresses the difficulty of training deep neural networks by setting initial weights to preserve signal variance across layers.

Glorot initialization defines weight distributions based on the number of incoming and outgoing connections of a layer, known as fan_in and fan_out. For a layer with these parameters, the weights are drawn so that the variance of activations remains approximately constant as information propagates through the network. There are two common variants:

- Glorot uniform: W is drawn from a uniform distribution with limits ±limit, where limit = sqrt(6 / (fan_in + fan_out)).

- Glorot normal: W is drawn from a normal distribution with zero mean and standard deviation std, where std = sqrt(2 / (fan_in + fan_out)).

Usage and impact: Glorot initialization has become a standard default in many deep learning frameworks because it helps mitigate vanishing and exploding gradients at initialization. It is widely implemented under names such as GlorotUniform and GlorotNormal, and is also known as Xavier initialization, reflecting the collaboration between Glorot and Bengio. While still popular for many activation functions, alternative initializations (for example, He initialization for ReLU-based networks) are used in specific contexts to optimize training dynamics.