Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

2026-06-21

AI Summary

This work addresses a key limitation in existing parameter-efficient fine-tuning methods for vision Transformers, which adapt each token independently and thereby ignore their intrinsic structural relationships, leading to redundant updates and spatial inconsistency. To overcome this, the authors propose performing adaptation in a hyperedge space: tokens are softly grouped into implicit hyperedges via a soft hypergraph construction, lightweight bottleneck adaptation is applied at the hyperedge level, and updates are propagated back to individual tokens through a hypergraph diffusion mechanism. This approach introduces structured hyperedge adaptation into parameter-efficient fine-tuning for the first time, explicitly modeling inter-token relationships and injecting structural inductive bias while preserving modularity and efficiency. Experiments demonstrate significant performance gains over state-of-the-art methods across multiple vision benchmarks, with particularly notable improvements on tasks requiring structural reasoning, underscoring the critical role of adaptation space design.

Abstract

Parameter-efficient fine-tuning (PEFT) has become a practical solution for adapting large pretrained vision transformers (ViTs) to downstream tasks while updating only a small subset of parameters. However, existing adapter-based methods perform adaptation independently for each token, implicitly assuming that token refinements should be learned in isolation. This token-wise formulation overlooks the structured relationships among tokens that naturally arise in visual scenes, potentially leading to redundant updates and spatially inconsistent feature refinement. In this work, we revisit the design of parameter-efficient adapters and propose to perform adaptation in hyperedge space rather than token space. We introduce HyperAdapter, a hypergraph-based adapter architecture that enables structured, group-aware adaptation through soft token routing. HyperAdapter constructs a soft hypergraph over ViT tokens using prototype-based assignments, aggregates token features into latent hyperedge representations, applies lightweight bottleneck adaptation at the hyperedge level, and diffuses the resulting updates back to tokens via the hypergraph incidence structure. This design injects an explicit structural inductive bias into PEFT while preserving the modularity and efficiency of standard adapters. Extensive experiments across diverse visual benchmarks demonstrate that structured hyperedge adaptation consistently outperforms strong PEFT baselines under comparable parameter budgets, with particularly pronounced gains on tasks requiring structured reasoning. Our results suggest that the choice of adaptation space is a critical yet underexplored dimension in parameter-efficient transfer for ViTs.

Problem

• parameter-efficient fine-tuning
• vision transformers
• structured relationships
• token adaptation
• hypergraph

Innovation

• Hypergraph-based adaptation
• Structured reasoning
• Parameter-efficient fine-tuning
• Vision Transformers
• Soft token routing