How Do Semantically Equivalent Code Transformations Impact Membership Inference on LLMs for Code?

2025-12-17

AI Summary

This work addresses the critical challenge of intellectual property (IP) compliance auditing in large language model (LLM) training for code. We systematically investigate how semantic-preserving code transformations affect the robustness of membership inference (MI) attacks. We propose a rule-based semantic-equivalence transformation framework and employ causal analysis—specifically do-calculus and perturbation analysis—to quantify the MI-disrupting effects of individual and combined transformations. Our study is the first to empirically demonstrate that variable renaming exhibits the strongest evasion capability, reducing MI attack success rate by 10.19%. Contrary to prevailing assumptions, composing multiple transformations yields no synergistic benefit. Crucially, transformed code retains strong fine-tuning utility, with only a 1.5% drop in downstream task accuracy. These findings expose a fundamental vulnerability in current open-source license compliance auditing pipelines and provide both theoretical grounding and empirical evidence for developing MI-resilient detection mechanisms and trustworthy code provenance systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance

Abstract

The success of large language models for code relies on vast amounts of code data, including public open-source repositories, such as GitHub, and private, confidential code from companies. This raises concerns about intellectual property compliance and the potential unauthorized use of license-restricted code. While membership inference (MI) techniques have been proposed to detect such unauthorized usage, their effectiveness can be undermined by semantically equivalent code transformation techniques, which modify code syntax while preserving semantic. In this work, we systematically investigate whether semantically equivalent code transformation rules might be leveraged to evade MI detection. The results reveal that model accuracy drops by only 1.5% in the worst case for each rule, demonstrating that transformed datasets can effectively serve as substitutes for fine-tuning. Additionally, we find that one of the rules (RenameVariable) reduces MI success by 10.19%, highlighting its potential to obscure the presence of restricted code. To validate these findings, we conduct a causal analysis confirming that variable renaming has the strongest causal effect in disrupting MI detection. Notably, we find that combining multiple transformations does not further reduce MI effectiveness. Our results expose a critical loophole in license compliance enforcement for training large language models for code, showing that MI detection can be substantially weakened by transformation-based obfuscation techniques.

Problem

• Investigates how code transformations evade membership inference detection
• Evaluates impact of semantic-preserving changes on model accuracy and MI success
• Exposes loopholes in license compliance for training code language models

Innovation

• Semantically equivalent transformations evade membership inference detection
• Variable renaming reduces membership inference success by 10.19%
• Transformed datasets substitute fine-tuning with minimal accuracy loss