Industrial-Scale Neural Network Clone Detection with Disk-Based Similarity Search

2025-04-24

AI Summary

Traditional k-NN–based code clone detection fails on billion-line industrial codebases due to severe memory constraints and pervasive approximate duplicate fragments that degrade retrieval accuracy. Method: We propose the first disk-native clone embedding indexing framework: (1) a Transformer encoder generates semantically robust code fragment embeddings; (2) an SSD-optimized external-memory index structure integrates enhanced disk-based approximate nearest neighbor (ANN) algorithms (e.g., DiskANN); and (3) a noise-resilient mechanism mitigates interference from massive approximate duplicates. Results: On codebases exceeding one billion lines, our approach achieves clone detection accuracy comparable to full-in-memory baselines, incurs only 2× SSD latency overhead, reduces storage footprint significantly, and improves query throughput by an order of magnitude. This work establishes the first high-accuracy, scalable, and industrially deployable neural code clone detection system.

Technology Category

• Search and Optimization: Distributed Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Web crawling and indexing
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Code clones are similar code fragments that often arise from copy-and-paste programming. Neural networks can classify pairs of code fragments as clone/not-clone with high accuracy. However, finding clones in industrial-scale code needs a more scalable approach than pairwise comparison. We extend existing neural network-based clone detection schemes to handle codebases that far exceed available memory, using indexing and search methods for external storage such as disks and solid-state drives. We generate a high-dimensional vector embedding for each code fragment using a transformer-based neural network. We then find similar embeddings using efficient multidimensional nearest neighbor search algorithms on external storage to find similar embeddings without pairwise comparison. We identify specific problems with industrial-scale code bases, such as large sets of almost identical code fragments that interact poorly with $k$-nearest neighbour search algorithms, and provide an effective solution. We demonstrate that our disk-based clone search approach achieves similar clone detection accuracy as an equivalent in-memory technique. Using a solid-state drive as external storage, our approach is around 2$ imes$ slower than the in-memory approach for a problem size that can fit within memory. We further demonstrate that our approach can scale to over a billion lines of code, providing valuable insights into the trade-offs between indexing speed, query performance, and storage efficiency for industrial-scale code clone detection.

Problem

• Scalable neural network detection for industrial code clones
• Efficient disk-based similarity search for large codebases
• Handling near-duplicate code fragments in nearest neighbor search

Innovation

• Transformer-based neural network for code embeddings
• Disk-based multidimensional nearest neighbor search
• Scalable solution for billion-line codebases