SimSiam

SimSiam, short for Simple Siamese Network for Contrastive Learning, is a self-supervised representation learning method for images introduced in 2020. It shows that a straightforward Siamese network can learn useful visual representations without relying on negative pairs, large batch sizes, or momentum encoders.

In training, two stochastic augmentations of the same image are created and passed through a shared network consisting of an encoder, a projection head, and a predictor head on one branch. The two branches share weights, forming a Siamese architecture. A key feature is the stop-gradient operation applied to the projection of one branch. The objective encourages the predictor output from one view to align with the projection of the other view, and vice versa, by maximizing the cosine similarity between cross-branch representations (with gradient flow blocked on the target branch). This setup helps prevent collapse to trivial solutions.

Typical architecture components include a convolutional encoder (often a ResNet), a small multilayer perceptron projection head, and a small multilayer perceptron predictor head. Unlike methods such as BYOL, SimSiam does not use a momentum (target) network, and it omits negative pairs and memory banks.

Training relies on standard data augmentations (e.g., random cropping, color jitter, flipping) and commonly uses stochastic optimization. Evaluation typically follows a linear-probe protocol, where a linear classifier is trained on frozen features from the pretrained encoder.

SimSiam has influenced the design of subsequent self-supervised methods by demonstrating that strong representations can be learned without negatives or momentum, highlighting the role of architectural and optimization tricks in preventing collapse.