MergeOver: Post-Training Token Merging for Recursive Vision Transformers

2026-08-13

AI Summary

This work addresses the persistent computational and memory overheads of recursive Vision Transformers, which remain challenging despite parameter reduction through weight sharing, and their incompatibility with token merging techniques without retraining. To overcome these limitations, we propose MergeOver—a post-training, plug-and-play method that, for the first time, effectively integrates token merging into the recursively weight-shared SReT architecture. Our approach resolves spatial misalignment and merging constraints via an unmerge tracking stack, safety-constrained merge ratio control, a cross-space permutation-aware token quality synchronization mechanism, and a staged single-pass scheduling strategy. Evaluated on ImageNet-1K, MergeOver incurs only a 1.47% accuracy drop while reducing GPU peak activation memory by 37.3% (batch=1) and 38.4% (batch=16), and decreases inference latency on Raspberry Pi 5 by 2.4% (batch=1) and 17.6% (batch=16).

Abstract

Vision Transformers (ViTs) demonstrate exceptional performance in computer vision but suffer from large parameter counts and quadratic computational complexity, severely limiting their deployment on resource-constrained edge hardware. While recursive weight-sharing reduces parameter counts and token merging mitigates computational and memory bottlenecks, integrating these two paradigms without costly retraining is non-trivial, leaving this intersection largely unexplored. We propose MergeOver, a post-training approach that integrates Token Merging (ToMe) into the recursively weight-shared Sliced Recursive Transformer (SReT). Through an Unmerge tracking stack, constraint-safe merge-rate adjustment, and synchronised token-mass tracking across spatial permutations, MergeOver resolves the spatial and merging constraints of this integration. We further employ a stage-wise single-shot schedule that performs token reduction at the first block of each stage and maintains a fixed sequence length throughout its subsequent recursive iterations. Benchmarked on ImageNet-1K, our selected configuration reduces top-1 accuracy by 1.47 percentage points. On the GPU, it reduces peak activation memory by 37.3% and 38.4% at batch sizes 1 and 16, while throughput decreases by 21.7% at batch size 1 but increases by 21.7% at batch size 16. On a Raspberry Pi 5 (ARM CPU), it reduces latency by 2.4% and 17.6% at batch sizes 1 and 16. These results show that MergeOver can recover a meaningful part of the throughput and memory cost that recursive weight-sharing introduces, without retraining, and provides a baseline for combining token merging with hierarchical recursive transformers.

Problem

• Vision Transformers
• recursive weight-sharing
• token merging
• post-training
• resource-constrained deployment

Innovation

• Token Merging
• Recursive Transformers
• Post-Training Compression
• Vision Transformers
• Memory Efficiency