Can Code Evaluation Metrics Detect Code Plagiarism?

2026-04-28

AI Summary

This study systematically evaluates the effectiveness of general-purpose code generation metrics for detecting code plagiarism across varying levels of complexity in software engineering education. Leveraging the ConPlag and IRPlag datasets, we compare five metrics—CodeBLEU, CrystalBLEU, RUBY, TSED, and CodeBERTScore—against established tools JPlag and Dolos under six increasingly sophisticated plagiarism scenarios. Our evaluation employs threshold-free ranking metrics that integrate semantic and structural similarity, incorporating techniques based on n-grams, abstract syntax trees, and pretrained language models. Results demonstrate that CrystalBLEU, when combined with appropriate preprocessing, outperforms Dolos overall and remains competitive even at the most challenging L6 level. Additionally, CodeBLEU and RUBY consistently surpass JPlag across most scenarios, underscoring the potential of general-purpose evaluation metrics in detecting sophisticated forms of code plagiarism.

Abstract

Source Code Plagiarism Detection (SCPD) plays an important role in maintaining fairness and academic integrity in software engineering education. Code Evaluation Metrics (CEMs) are developed for assessing code generation tasks. However, it remains unclear whether such metrics can reliably detect plagiarism across different levels of modification (L1-L6), increasing in complexity. In this paper, we perform a comparative empirical study using two open-source labelled datasets, ConPlag (raw and template-free versions) and IRPlag. We evaluate five CEMs, namely CodeBLEU, CrystalBLEU, RUBY, Tree Structured Edit Distance (TSED), and CodeBERTScore. The performance is evaluated using threshold-free ranking-based measures to assess overall, per dataset, and per-level plagiarism performance. The results are compared against state-of-the-art (SOTA) Source Code Plagiarism Detection Tools (SCPDTs), JPlag and Dolos. Our findings show that without preprocessing, Dolos achieves the highest overall ranking performance, while among the individual metrics, CrystalBLEU, CodeBLEU, and RUBY outperform JPlag. Performance is strongest at L1 and drops from L4 onward, while CrystalBLEU remains competitive on L6. With preprocessing, CrystalBLEU surpasses Dolos overall. Per dataset, Dolos achieved the best ranking on the ConPlag raw dataset, while CrystalBLEU was the best-performing metric on the remaining datasets. At the plagiarism levels, Dolos remains strongest on L4, while Crystal-BLEU leads most of the remaining difficult levels. These results indicate that CEMs are comparable to dedicated tools in terms of ranking metrics.

Problem

• Source Code Plagiarism Detection
• Code Evaluation Metrics
• Plagiarism Levels
• Code Similarity
• Academic Integrity

Innovation

• Code Evaluation Metrics
• Source Code Plagiarism Detection
• CrystalBLEU
• Empirical Study
• Ranking-based Evaluation