CodeVision: Detecting LLM-Generated Code Using 2D Token Probability Maps and Vision Models

2025-01-06

AI Summary

To address the academic integrity crisis in educational settings arising from the indistinguishability of human-written and large language model (LLM)-generated code (e.g., from ChatGPT), this paper proposes a novel vision-based detection method. The core innovation lies in transforming source code into 2D logarithmic probability maps that preserve spatial structures—including indentation, bracket nesting, and layout—thereby modeling code as visual input for the first time. Leveraging vision transformers (ViT) and ResNet architectures, the approach jointly captures syntactic, semantic, and structural features, overcoming limitations of conventional sequence-based text models. Extensive experiments demonstrate high accuracy, strong cross-language generalization, and excellent scalability across diverse programming languages. The method significantly enhances robustness and practicality for detecting LLM-generated code in real-world educational applications.

Technology Category

• Computer Vision: Large Vision Models
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rise of large language models (LLMs) like ChatGPT has significantly improved automated code generation, enhancing software development efficiency. However, this introduces challenges in academia, particularly in distinguishing between human-written and LLM-generated code, which complicates issues of academic integrity. Existing detection methods, such as pre-trained models and watermarking, face limitations in adaptability and computational efficiency. In this paper, we propose a novel detection method using 2D token probability maps combined with vision models, preserving spatial code structures such as indentation and brackets. By transforming code into log probability matrices and applying vision models like Vision Transformers (ViT) and ResNet, we capture both content and structure for more accurate detection. Our method shows robustness across multiple programming languages and improves upon traditional detectors, offering a scalable and computationally efficient solution for identifying LLM-generated code.

Problem

• Education Environment
• Code Originality
• Cheating Prevention

Innovation

• CodeVision
• Image Recognition Models
• Efficient Code Analysis