Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation

2026-08-08

AI Summary

This work addresses the computational inefficiency of existing parameter-efficient fine-tuning (PEFT) methods, which, despite reducing the number of trainable parameters, still require extensive training steps. The authors propose a novel approach that introduces circuit discovery techniques during the module selection phase prior to fine-tuning. By employing near-zero-initialized probe heads, the method isolates the backbone network’s response to the target task, thereby eliminating interference from classifier bias. Only the identified critical subgraph modules are subsequently fine-tuned. Notably, this strategy eliminates the need for learning rate warm-up and converges in approximately 20 training epochs on average. Without increasing model parameters or inference overhead, the method reduces FLOPs by 2.3–6.6× compared to mainstream PEFT approaches and achieves up to a 16× reduction in actual training time.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has become the de facto standard for adapting Vision Transformers (ViTs) to downstream tasks. While parameter count has been the dominant efficiency metric in PEFT, it does not imply \textit{compute efficiency}: parameter-sparse methods can still incur full-model training cost per step, and typically need long schedules to reach peak accuracy. We introduce Circuit Fine-Tuning (CFT), a compute-efficient framework that uses circuit discovery---conventionally used to explain trained models---to select modules for fine-tuning before training. Whereas attribution is conventionally formulated against a trained task head, we formulate it against a near-zero-initialized probe head, which isolates the response of the backbone to the target distribution rather than the preferences of a particular classifier. CFT then fine-tunes only the recovered subgraph. CFT needs no learning-rate warmup and reaches peak accuracy in ${\sim}20$ epochs on average---versus $44$--$96$ for strong PEFT baselines---yielding $2.3$--$6.6\times$ fewer training FLOPs and up to $16\times$ less wall-clock time, while adding zero parameters and no inference operations. Experiments across a standard visual transfer benchmark (VTAB-1k), hierarchical backbones (Swin), domain-shifted medical imaging (CBIS-DDSM), and a vision-language model (Gemma-3 on CUB-200) demonstrate the effectiveness of CFT. Code is available at https://github.com/UriKialy/CFT

Problem

• Parameter-Efficient Fine-Tuning
• Compute Efficiency
• Vision Transformers
• Training FLOPs
• Fine-Tuning

Innovation

• Circuit Fine-Tuning
• Compute-Efficient Adaptation
• Parameter-Efficient Fine-Tuning
• Circuit Discovery
• Vision Transformers