DEIT
DeiT, short for Data-efficient Image Transformer, is a family of vision transformer models and training techniques introduced by Facebook AI Research in 2020. It seeks to make training Vision Transformers (ViT) more data-efficient, enabling competitive performance on large-scale image recognition tasks with less labeled data and compute than previous ViT approaches.
The core idea of DeiT is to combine a standard Vision Transformer with knowledge distillation from a
Regularization and data augmentation play important roles in DeiT. The method employs strong augmentation strategies and
Impact-wise, DeiT catalyzed interest in data-efficient training for transformers and spurred further research into distillation-based supervision