Tidsformers

Tidsformers is a proposed class of neural network models that extend the transformer architecture with explicit temporal modeling. The term blends "tid" (time) with "formers." In this concept, time is treated as a first-class dimension alongside sequence position, enabling reasoning across multiple timelines and irregular intervals.

Core ideas include time-aware attention that can attend over content and time, and a temporal memory storing past representations with timestamps. Time embeddings encode absolute or relative timing and can be conditioned on sampling rates or time gaps. Some designs separate temporal axes or use continuous time encodings.

Architectural variants span encoder-decoder models for forecasting and auto-regressive forms for sequential generation. Training may add losses for temporal consistency, such as predicting future steps or reconstructing missing times. Efficiency methods include sparse attention, multi-scale time windows, and memory compression to manage long histories.

Applications discussed in the literature include irregularly sampled time-series forecasting, event streams, climate and ecology modeling, finance, and healthcare monitoring. By modeling time more explicitly, tidsformers aim to improve long-range dependency handling and reasoning about temporal causality and patterns beyond fixed position bounds.

Limitations include increased complexity and data requirements, potential overfitting to temporal cues, and evaluation challenges due to varying time granularities. As a relatively new idea, tidsformers remain exploratory, with ongoing work on time embeddings, memory mechanisms, and scalable training methods.