Invariant Pretraining for Robust Code Representations

2026-08-15

AI Summary

This study addresses the insufficient robustness of code encoders to semantically equivalent yet syntactically distinct programs by proposing a pure-code multi-positive invariant contrastive learning paradigm. By integrating semantics-preserving transformations with a hybrid contrastive strategy, this approach enhances model invariance to syntactic variants without requiring natural language data. Experimental results demonstrate that the proposed method improves robustness by 11% and 19% on clone detection and code classification tasks, respectively, while maintaining or surpassing standard performance. Furthermore, ablation studies confirm that multi-positive invariant contrastive learning serves as the core contribution, effectively bridging the robustness gap in existing code encoders.

Abstract

Encoder-based code representation models remain widely deployed for discriminative tasks such as clone detection and code classification, where their small size and low inference cost are decisive. Their robustness, however, is fragile: under invariant programs, semantically equivalent code written in different syntactic forms, learned representations degrade substantially even though program behavior is unchanged. We present an empirical study of this robustness gap across four encoder baselines, two downstream tasks, and four datasets, together with a minimal code-only continued pretraining recipe that closes much of it. Our method, invariant pretraining (InvPT), applies semantic-preserving transformations to the corpus and combines masked language modeling with multi-positive supervised contrastive learning that treats all augmentations of the same source function as positives, mixing self-contrast pairs (same code, different masks) with invariant-contrast pairs (transformed code) for positives of varying difficulty. Unlike prior contrastive code encoders, InvPT does not require paired natural-language data. Across our evaluation, InvPT improves robustness on transformed test sets by up to 11 percentage points on clone detection and 19 on code classification while matching or improving standard accuracy, and our ablations isolate multi-positive invariant contrast as the main source of the gains. Our aim is not a new objective but a careful measurement of where encoder robustness breaks and how far a simple, code-only recipe can recover it.

Problem

• Code Representation
• Robustness
• Invariant Programs
• Encoder-based Models
• Semantic Equivalence

Innovation

• Invariant Pretraining
• Multi-positive Supervised Contrastive Learning
• Semantic-preserving Transformations
• Code-only Pretraining
• Robust Code Representations