ViViT

ViViT, short for Video Vision Transformer, is a family of transformer-based architectures for video understanding, particularly action recognition. It extends the Vision Transformer (ViT) approach from images to video by applying patch-based representations to individual frames and modeling temporal dynamics with transformer layers. ViViT was introduced by Arnab, De, et al. in 2021, presenting a scalable framework for processing video data with self-attention mechanisms.

The core idea of ViViT is to treat a video clip as a sequence of tokens derived from frame patches. Each frame is divided into non-overlapping patches, projected linearly, and augmented with positional embeddings. These tokens are fed into a Transformer encoder to produce a representation of the video. To manage the computational cost of spatiotemporal attention, ViViT proposes factorized space-time attention in some variants, which performs spatial attention within frames followed by temporal attention across frames. A joint space-time attention variant computes attention over the full spatiotemporal token set. ViViT configurations vary by patch size, number of frames, and model scale, with base and large variants and options for pretraining on large image or video collections.

ViViT has been evaluated on standard benchmarks such as Kinetics-400 and Kinetics-700 and has shown competitive performance with CNN-based architectures when trained with sufficient data. The approach demonstrated the viability of transformers for video understanding and influenced subsequent models in the field, contributing to the development of TimeSformer and MViT. ViViT’s design emphasizes capturing both appearance and motion cues through self-attention over spatiotemporal tokens, enabling end-to-end learning without reliance on conventional 3D convolutions.