Depotformer

Depotformer is a class of neural network architectures that augments transformer models with an external, trainable memory component often referred to as a "depot." The depot stores a fixed or adaptive set of memory vectors that can be read from and written to by standard attention mechanisms, enabling models to retain and retrieve information across longer contexts or multiple tasks without extending the token sequence length.

Architectural variants typically interleave transformer layers with depot access operations: input tokens attend to depot vectors via cross-attention to enrich representations, while depot vectors are updated by attended summaries of recent inputs. Implementations may use sparse attention, routing, or gating to limit computation and control which memory slots are read or modified. Training objectives can combine task losses with auxiliary reconstruction or contrastive terms to stabilize memory contents.

Depotformer-style designs are explored for applications that benefit from persistent or compacted knowledge, including long-context language modeling, retrieval-augmented generation, continual learning, multi-modal fusion, and episodic reasoning in robotics. Advantages include improved effective context length, parameter efficiency for repeated information, and a structured way to incorporate stored facts. Known limitations are capacity constraints, potential memory drift or interference, added training complexity, and sensitivity to how memory is allocated and updated.

Depotformer approaches relate to memory networks, retrieval-augmented models, and compressive or recurrent transformer variants. Research is active on memory management strategies, interpretability of depot contents, and scaling trade-offs between depot size and model depth.