🤖 AI Summary
Vision Transformers suffer from phase dependency in dense prediction tasks due to their fixed image patch grid, causing pixel-level outputs to vary unstably near patch boundaries. This work formally characterizes patch phase as a measurable nuisance variable and introduces Uniform Phase Marginalization—a plug-and-play, test-time post-processing method that requires no retraining. By performing forward inference across multiple structured phase shifts, inversely aligning the predictions, and aggregating them in the original image coordinate system, the method substantially improves prediction consistency. On Cityscapes, it surpasses the strongest four-shift test-time augmentation by 0.31 mIoU at comparable computational cost; further increasing the number of phases (K=8/16) yields only marginal gains while significantly increasing latency.
📝 Abstract
Vision Transformers operate on fixed patch grids, which can introduce phase-dependent instability for dense prediction: changing the patch partition can change the token evidence available to a pixel, especially near boundaries. We formalize patch-grid phase as a nuisance variable and propose Phase Marginalization, a post-hoc marginalization method that evaluates structured patch-grid phases, inverse-aligns dense outputs, and aggregates them in the original image coordinate system. The central variant, Uniform Phase Marginalization with K = 4, is training-free and improves over the canonical K = 1 baseline across measured segmentation, depth, and local matching settings. In a controlled Cityscapes experiment, Uniform Phase Marginalization provides a modest compute-matched advantage over generic shift-based four-forward test-time augmentation (TTA) (+0.31 mean Intersection-over-Union over the strongest tested generic row). A scaling study further shows that K = 4 is a practical cost-accuracy trade-off: K = 8 is essentially unchanged and K = 16 adds little accuracy at much higher latency. These results position patch-grid phase as a measurable nuisance variable and Phase Marginalization as a simple diagnostic and post-hoc marginalization baseline for dense ViT prediction.