MTFer

MTFer is a term used to describe a family of multitask transformer architectures designed to efficiently learn and perform multiple tasks within a single model. The approach extends standard transformer models by incorporating lightweight, task-specific adapters that modulate the shared representation and by leveraging task tokens or prompts to condition the model on the current task.

The core idea behind MTFer is to maintain a common, powerful backbone while allowing small, task-specific adjustments. Adapters are small neural modules inserted after transformer layers; they learn residual transformations that tailor the shared features to individual tasks without rewriting the entire network. Task conditioning can be achieved with a learned task embedding, a task prefix, or a soft prompt, guiding the model toward task-specific outputs while preserving a common representation for knowledge sharing.

During training, a multitask objective combines losses from all tasks, often with balancing strategies to prevent any single task from dominating. Some implementations explore cross-task attention to enable information flow between tasks more directly, enhancing generalization across related objectives. Inference uses the same backbone across tasks, with the chosen conditioning mechanism directing the final task head, which can be a classifier, regressor, or sequence generator.

Variants of MTFer emphasize different priorities. MTFer-Lite focuses on parameter efficiency with compact adapters; MTFer-Plus adds explicit cross-task attention modules; MTFer-Continual targets continual learning, aiming to preserve performance on earlier tasks while incorporating new ones. Limitations include potential negative transfer, task interference, and memory overhead from maintaining multiple adapters.

Applications span natural language processing, vision-language tasks, and cross-domain adaptation, particularly where a single system handles multiple related objectives. See also multitask learning, adapters, transformer architectures, and prompt-based conditioning.