TimeSformer

TimeSformer is a neural network architecture designed for video understanding that applies the Transformer framework to spatiotemporal data. It extends the concept of Vision Transformers (ViT) to video by representing a clip as a sequence of tokens formed from patches sampled across multiple frames. Each frame is divided into non-overlapping patches, which are flattened and projected to a latent dimension. A class token and learnable spatial and temporal position embeddings are added, enabling the model to incorporate both where a patch is located and when it appears in the sequence.

A central contribution of TimeSformer is the idea of Divided Space-Time Attention, an efficient alternative to full spatiotemporal attention. Rather than computing attention across all space-time tokens in a single operation, TimeSformer alternates between temporal attention (across the time axis for each spatial location) and spatial attention (across the spatial patches within a frame). This factorization reduces computational cost and memory usage, making the model more scalable for video data. Variants of the approach include using temporal-first and spatial-first attention layers, as well as configurations that apply a joint, all-tokens-attend-to-all-tokens attention in some blocks.

In evaluation, TimeSformer variants demonstrated competitive performance on standard video action recognition benchmarks, achieving strong results with lower computational requirements than some prior video transformer models. The approach highlighted the viability of Transformer-based architectures for video and influenced subsequent work aimed at efficient spatiotemporal modeling.

Limitations cited in early work include sensitivity to pretraining data size and potential challenges in capturing very long-range temporal dependencies, motivating ongoing research into scaling, pretraining strategies, and integration with multimodal information.