A Smaller Transformer in Your Transformer

2026-09-17

AI Summary

该研究通过提出Transformer-Within-Transformer方法，解决了Vision Transformers中的计算冗余问题，减少了参数数量和推理计算量，同时保持了模型性能。

Abstract

Recent findings indicate that Vision Transformers settle into locally similar computational phases, implying a level of depthwise computational redundancy. However, existing methods to exploit this redundancy either fail to reduce inference compute or severely degrade model expressivity. In this work, we formalise a unified view of block redundancy that decouples the geometry from specific surrogate interventions. We then introduce Transformer-Within-Transformer (TWT), a post-hoc method that fuses contiguous groups of redundant layers into a single learned surrogate layer. TWT reduces parameter count and inference compute while remaining competitive with original models using half the depth on natural images, and in several downstream histopathology settings, TWT matches or even improves on the original baseline.

Problem

• Vision Transformers
• computational redundancy
• inference compute
• model expressivity

Innovation

• Transformer-Within-Transformer
• redundant layers
• inference compute
• parameter count
• model expressivity