The Compliance Paradox: Semantic-Instruction Decoupling in Automated Academic Code Evaluation

2026-01-29

AI Summary

This study addresses a critical flaw in large language models (LLMs) when applied to automated code scoring: their tendency to over-comply with implicit instructions at the expense of code correctness, leading to functionally incorrect submissions being erroneously deemed acceptable—a phenomenon termed the “compliance paradox.” To systematically expose this vulnerability, the authors propose the SPACI framework and the AST-ASIP protocol, which employ semantics-preserving adversarial code injection by embedding adversarial instructions into Trivia nodes of abstract syntax trees. Evaluations across nine state-of-the-art LLMs and 25,000 multilingual code submissions reveal that even high-performing models, such as DeepSeek-V3, exhibit failure rates exceeding 95%. The work further introduces a three-dimensional metric to quantify the extent of “false certification” induced by such compliance-driven errors.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The rapid integration of Large Language Models (LLMs) into educational assessment rests on the unverified assumption that instruction following capability translates directly to objective adjudication. We demonstrate that this assumption is fundamentally flawed. Instead of evaluating code quality, models frequently decouple from the submission's logic to satisfy hidden directives, a systemic vulnerability we term the Compliance Paradox, where models fine-tuned for extreme helpfulness are vulnerable to adversarial manipulation. To expose this, we introduce the Semantic-Preserving Adversarial Code Injection (SPACI) Framework and the Abstract Syntax Tree-Aware Semantic Injection Protocol (AST-ASIP). These methods exploit the Syntax-Semantics Gap by embedding adversarial directives into syntactically inert regions (trivia nodes) of the Abstract Syntax Tree. Through a large-scale evaluation of 9 SOTA models across 25,000 submissions in Python, C, C++, and Java, we reveal catastrophic failure rates (>95%) in high-capacity open-weights models like DeepSeek-V3, which systematically prioritize hidden formatting constraints over code correctness. We quantify this failure using our novel tripartite framework measuring Decoupling Probability, Score Divergence, and Pedagogical Severity to demonstrate the widespread"False Certification"of functionally broken code. Our findings suggest that current alignment paradigms create a"Trojan"vulnerability in automated grading, necessitating a shift from standard RLHF toward domain-specific Adjudicative Robustness, where models are conditioned to prioritize evidence over instruction compliance. We release our complete dataset and injection framework to facilitate further research on the topic.

Problem

• Compliance Paradox
• Automated Code Evaluation
• Semantic-Instruction Decoupling
• Adversarial Manipulation
• False Certification

Innovation

• Compliance Paradox
• Semantic-Preserving Adversarial Code Injection
• Abstract Syntax Tree-Aware Semantic Injection
• Adjudicative Robustness
• Syntax-Semantics Gap