dropconnect

DropConnect is a regularization technique used in neural networks that randomly drops connections by zeroing individual weights during training. Unlike dropout, which randomly drops activations, DropConnect applies the stochastic masking directly to the weight matrix.

Mechanistically, during training each weight is kept with probability p and set to zero with probability 1-p for a given forward and backward pass. This creates an ensemble of thinned networks, reducing co-adaptation of features and helping to prevent overfitting. The masking is typically re-sampled for each training pass or mini-batch.

For inference, the stochastic masking is not usually applied. Instead, the expected effect of the masks is used, commonly by scaling the weight matrix by p to approximate the averaged contribution of the thinned networks. Some implementations may use Monte Carlo estimation with multiple forward passes to average predictions.

DropConnect is applicable to any layer that uses weights, including fully connected layers and convolutional layers (where it masks the convolutional kernels). It can be used alone or in combination with other regularization methods such as dropout and weight decay. The key practical consideration is the keep probability p; overly aggressive masking can hinder learning, while too little masking may provide limited regularization.

Relation to other methods: DropConnect shares the goal of reducing overfitting with dropout and other stochastic regularizers, but targets weights instead of activations. It is discussed in the literature as an alternative approach to introducing noise into the training process and promoting robust feature learning.