SemRep: Generative Code Representation Learning with Code Transformations

2026-03-13

AI Summary

Existing code translation approaches are hindered by end-to-end learning paradigms or rigid compiler abstractions, limiting their ability to construct high-quality semantic representations. This work proposes a generative code representation learning framework that, for the first time, leverages semantics-preserving code transformations as an intermediate task and guiding signal, substantially enhancing the generalization, robustness, and optimization capabilities of learned representations while enabling efficient evolutionary search. Experimental results demonstrate that, under identical training budgets, the proposed method significantly outperforms strong baselines in correctness (+6.9%), performance (+1.1×), generalization (+13.9%), and robustness (+6.7%). Moreover, when integrated with an evolutionary agent, it discovers superior optimization solutions—unattainable by larger models—using only 75% of the inference compute.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Representation Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Code transformation is a foundational capability in the software development process, where its effectiveness relies on constructing a high-quality code representation to characterize the input code semantics and guide the transformation. Existing approaches treat code transformation as an end-to-end learning task, leaving the construction of the representation needed for semantic reasoning implicit in model weights or relying on rigid compiler-level abstractions. We present SemRep, a framework that improves code transformation through generative code representation learning. Our key insight is to employ the semantics-preserving transformations as the intermediate representation, which serves as both a generative mid-training task and the guidance for subsequent instruction-specific code transformations. Across general code editing and optimization tasks (e.g., GPU kernel optimization), SemRep outperforms the extensively finetuned baselines with strictly the same training budget by 6.9% in correctness, 1.1x in performance, 13.9% in generalization, and 6.7% in robustness. With the improved exploration of diverse code transformations, SemRep is particularly amenable to evolutionary search. Combined with an evolutionary coding agent, SemRep finds optimizations that 685B larger-weight baselines fail to discover while achieving the same performance with 25% less inference compute.

Problem

• code transformation
• code representation
• semantic reasoning
• generative learning
• program optimization

Innovation

• generative code representation
• semantics-preserving transformation
• code transformation
• evolutionary search
• intermediate representation