CodeIP: A Grammar-Guided Multi-Bit Watermark for Large Language Models of Code

2024-04-24

Conference on Empirical Methods in Natural Language Processing

AI Summary

To address the lack of traceable identifiers in large language model (LLM)-generated code—hindering intellectual property protection and academic integrity detection—this paper proposes a syntax-aware multi-bit watermarking mechanism. Methodologically, it introduces the first AST-guided token sampling constraint integrated with a type predictor for controlled decoding, enabling embedding of multi-dimensional provenance information (e.g., model vendor). A robust encoding and extraction algorithm is designed to ensure high watermark detection accuracy and strong resilience against semantic-preserving perturbations. The key contribution lies in overcoming the conventional single-bit watermarking limitation by simultaneously maximizing information capacity and preserving syntactic correctness. Extensive experiments across five real-world programming language datasets demonstrate that the method achieves >98% watermark detection accuracy while maintaining 100% syntactic validity of generated code—significantly outperforming existing baselines.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have achieved remarkable progress in code generation. It now becomes crucial to identify whether the code is AI-generated and to determine the specific model used, particularly for purposes such as protecting Intellectual Property (IP) in industry and preventing cheating in programming exercises. To this end, several attempts have been made to insert watermarks into machine-generated code. However, existing approaches are limited to inserting only a single bit of information. In this paper, we introduce CodeIP, a novel multi-bit watermarking technique that inserts additional information to preserve crucial provenance details, such as the vendor ID of an LLM, thereby safeguarding the IPs of LLMs in code generation. Furthermore, to ensure the syntactical correctness of the generated code, we propose constraining the sampling process for predicting the next token by training a type predictor. Experiments conducted on a real-world dataset across five programming languages demonstrate the effectiveness of CodeIP in watermarking LLMs for code generation while maintaining the syntactical correctness of code.

Problem

• Intellectual Property Protection
• Code Tracing
• AI Model Identification

Innovation

• CodeIP
• Intellectual Property Protection
• Complex Information Embedding