Model Compression vs. Adversarial Robustness: An Empirical Study on Language Models for Code

2025-08-05

AI Summary

This study investigates the impact of model compression—specifically pruning, quantization, and knowledge distillation—on the adversarial robustness of Transformer-based code language models. Addressing the critical issue of significantly degraded robustness in compressed models under security-sensitive scenarios, we conduct the first empirical evaluation in the code domain, assessing three state-of-the-art code models under four canonical adversarial attacks using six-dimensional robustness metrics. Results show that while compressed models retain strong performance on standard tasks, their adversarial accuracy drops by an average of 23.6%, exposing a fundamental trade-off between model compactness and security. We further analyze the distinct mechanistic effects of each compression strategy on robustness, revealing how architectural and representational changes propagate vulnerability. This work provides both theoretical insights and practical guidelines for designing and deploying lightweight, security-aware code models in software engineering applications.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Transformer-based language models for code have shown remarkable performance in various software analytics tasks, but their adoption is hindered by high computational costs, slow inference speeds, and substantial environmental impact. Model compression techniques such as pruning, quantization, and knowledge distillation have gained traction in addressing these challenges. However, the impact of these strategies on the robustness of compressed language models for code in adversarial scenarios remains poorly understood. Understanding how these compressed models behave under adversarial attacks is essential for their safe and effective deployment in real-world applications. To bridge this knowledge gap, we conduct a comprehensive evaluation of how common compression strategies affect the adversarial robustness of compressed models. We assess the robustness of compressed versions of three widely used language models for code across three software analytics tasks, using six evaluation metrics and four commonly used classical adversarial attacks. Our findings indicate that compressed models generally maintain comparable performance to their uncompressed counterparts. However, when subjected to adversarial attacks, compressed models exhibit significantly reduced robustness. These results reveal a trade-off between model size reduction and adversarial robustness, underscoring the need for careful consideration when deploying compressed models in security-critical software applications. Our study highlights the need for further research into compression strategies that strike a balance between computational efficiency and adversarial robustness, which is essential for deploying reliable language models for code in real-world software applications.

Problem

• Impact of model compression on adversarial robustness in code language models
• Trade-off between model size reduction and adversarial attack resilience
• Need for balanced compression strategies ensuring efficiency and robustness

Innovation

• Evaluates compression impact on adversarial robustness
• Tests three models with six metrics, four attacks
• Reveals trade-off between compression and robustness