DEIT

DeiT, short for Data-efficient Image Transformer, is a family of vision transformer models and training techniques introduced by Facebook AI Research in 2020. It seeks to make training Vision Transformers (ViT) more data-efficient, enabling competitive performance on large-scale image recognition tasks with less labeled data and compute than previous ViT approaches.

The core idea of DeiT is to combine a standard Vision Transformer with knowledge distillation from a convolutional neural network teacher. During training, a distillation token is appended to the transformer’s input sequence and learns from both the true labels and the teacher’s soft predictions. A teacher model, typically a pretrained CNN such as ResNet-50, provides guidance through a distillation loss that complements the conventional cross-entropy loss. This setup helps the student ViT learn robust representations even when data is limited.

Regularization and data augmentation play important roles in DeiT. The method employs strong augmentation strategies and careful optimization (for example, AdamW) to improve generalization. DeiT demonstrates that ViTs can be trained from scratch on ImageNet with competitive accuracy when aided by distillation and augmentation, and it introduces several model sizes to balance accuracy and computational cost (for example, DeiT-Tiny, DeiT-Small, and DeiT-Base).

Impact-wise, DeiT catalyzed interest in data-efficient training for transformers and spurred further research into distillation-based supervision for vision models. It showed that transformer architectures could reach strong performance with less data than previously required, broadening their practical applicability in image recognition tasks.