Machine vs Machine: Using AI to Tackle Generative AI Threats in Assessment

2025-05-31

AI Summary

Generative AI models (e.g., GPT-4, Claude, Llama) produce high-quality academic content, severely undermining the validity and credibility of traditional higher education assessments; existing detection tools and manual redesign approaches suffer from bias, subjectivity, and poor scalability. Method: We propose a systematic “machine-vs-machine” framework featuring a novel dual-track assessment vulnerability theory: (1) a static track distills eight anti-generation design principles grounded in fundamental AI capability bottlenecks; (2) a dynamic track establishes a simulation-based adversarial testing mechanism. We further develop an interpretable, weighted, threshold-defined quantitative scoring model to precisely identify assessment vulnerabilities and enhance robustness. Contribution/Results: This framework provides the first systematic theoretical foundation and methodological toolkit for building credible, robust, and scalable educational assessments in the AI era—enabling precise vulnerability detection and resilience improvement through principled, empirically grounded design.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

This paper presents a theoretical framework for addressing the challenges posed by generative artificial intelligence (AI) in higher education assessment through a machine-versus-machine approach. Large language models like GPT-4, Claude, and Llama increasingly demonstrate the ability to produce sophisticated academic content, traditional assessment methods face an existential threat, with surveys indicating 74-92% of students experimenting with these tools for academic purposes. Current responses, ranging from detection software to manual assessment redesign, show significant limitations: detection tools demonstrate bias against non-native English writers and can be easily circumvented, while manual frameworks rely heavily on subjective judgment and assume static AI capabilities. This paper introduces a dual strategy paradigm combining static analysis and dynamic testing to create a comprehensive theoretical framework for assessment vulnerability evaluation. The static analysis component comprises eight theoretically justified elements: specificity and contextualization, temporal relevance, process visibility requirements, personalization elements, resource accessibility, multimodal integration, ethical reasoning requirements, and collaborative elements. Each element addresses specific limitations in generative AI capabilities, creating barriers that distinguish authentic human learning from AI-generated simulation. The dynamic testing component provides a complementary approach through simulation-based vulnerability assessment, addressing limitations in pattern-based analysis. The paper presents a theoretical framework for vulnerability scoring, including the conceptual basis for quantitative assessment, weighting frameworks, and threshold determination theory.

Problem

• Addressing generative AI threats in higher education assessment
• Overcoming limitations of current AI detection and manual methods
• Proposing a dual strategy for evaluating assessment vulnerability

Innovation

• Dual strategy combining static and dynamic analysis
• Eight-element static framework targeting AI limitations
• Dynamic simulation-based vulnerability assessment approach