RepTran: Search-Based Repair of Transformer Models

2026-07-13

AI Summary

Transformer models are widely deployed in AI systems, yet they lack targeted mechanisms for correcting erroneous behaviors. This work proposes RepTran—the first repair method specifically designed for Transformer architectures—focusing on the feed-forward network (FFN) modules. RepTran identifies critical parameters by combining variance-based neuron scoring with bidirectional suspicious weight scoring, and iteratively refines them using a differential evolution algorithm. By tailoring the repair process to the structural characteristics of Transformers, RepTran overcomes the limitations of generic deep neural network repair strategies. Evaluated on 18 fault benchmarks constructed from CIFAR-100 and Tiny-ImageNet, the method achieves an average repair rate of 74.7%, substantially outperforming baseline approaches including random selection, Arachne, and ArachneW.

Abstract

To ensure the overall quality of AI-enabled software, not only traditional software components but also AI components need to be tested and repaired. Among AI components, Transformer models are increasingly integrated into software systems, which makes their misbehaviors critical. Although prior work in the software engineering community has proposed deep neural network (DNN) repair methods, most overlook Transformer-specific structures. We propose RepTran, a search-based repair method for Transformer models. It targets their feed-forward networks (FFNs), which play a central role in the architecture. RepTran identifies suspicious weights by combining two types of scores: a variance-based neuron score and an existing bidirectional score. It then iteratively optimizes these weights using differential evolution. Our evaluation includes 18 fault benchmarks constructed from CIFAR-100 and Tiny-ImageNet. We compare RepTran against three baselines: random weight selection, Arachne (a state-of-the-art DNN repair method), and ArachneW, which enables Arachne to control the number of selected weights. RepTran achieved an average repair rate of 74.7%, statistically outperforming random selection and Arachne across all benchmarks. Effect size analysis revealed that RepTran achieved higher repair rates than ArachneW regardless of the number of selected weights. These results suggest that RepTran is effective for enhancing the reliability of AI-enabled software.

Problem

• Transformer models
• model repair
• AI-enabled software
• fault repair
• deep neural networks

Innovation

• Transformer repair
• search-based optimization
• feed-forward network
• differential evolution
• neuron suspiciousness scoring