🤖 AI Summary
This work addresses the significant performance gap of small-scale Transformers—compared to LSTMs and MLPs—under task-switching scenarios (e.g., dynamically triggered incremental, additive, reverse-copy, and context operations via control tokens). To this end, we propose Cisformer, a novel architecture incorporating non-translation-invariant positional encodings to better capture control-token semantics, and Extensive Attention, which expands the attention scope and explicitly models task-switching logic. Evaluated on our newly constructed, bounded-domain arithmetic benchmark IARC, the Cisformer–Extensive Attention combination achieves ≈95% accuracy—substantially surpassing standard Transformers (<50%), LSTMs (≈70%), and MLPs (≈70%). Our findings demonstrate that embedding structured inductive biases into attention mechanisms can effectively overcome the performance bottlenecks of lightweight sequence models in dynamic task-switching settings, establishing a new paradigm for task-adaptive design of resource-efficient architectures.
📝 Abstract
The rapid progress seen in terms of large-scale generative AI is largely based on the attention mechanism. It is conversely non-trivial to conceive small-scale applications for which attention-based architectures outperform traditional approaches, such as multi-layer perceptrons or recurrent networks. We examine this problem in the context of 'task switching'. In this framework models work on ongoing token sequences with the current task being determined by stochastically interspersed control tokens. We show that standard transformers cannot solve a basic task switching reference model based on finite domain arithmetics which contains subtasks dedicated to increment / addition / reverse copy / context (IARC). We show that transformers, long short-term memory recurrent networks (LSTM), and plain multi-layer perceptrons (MLPs) achieve similar, but only modest prediction accuracies. We enlarge our comparative study by including an extension of the standard transformer architecture to its non-translational invariant counterpart, the cisformer, and an alternative attention mechanism, extensive attention. A combination of the latter is found to be the only model able to achieve considerable performance levels, of around 95%. Our results indicate that the workings of attention can be understood better, and even improved, when comparing qualitatively different formulations in task-switching settings.