Process Matters more than Output for Distinguishing Humans from Machines

2026-05-07

AI Summary

Current approaches that rely solely on output behavior struggle to reliably distinguish humans from large language models, particularly when model outputs are highly human-like. This work proposes shifting the focus from behavioral outcomes to underlying cognitive processes, introducing CogCAPTCHA30—a benchmark comprising 30 carefully designed tasks—to systematically evaluate the discriminative power of process-based features. The study demonstrates that cognitive process signatures consistently outperform task performance metrics in identifying human versus model agents, achieving an average AUC of 0.88 through cognitive task design, process feature extraction, and classifier evaluation—even under output-matching conditions. Furthermore, the authors propose Process-level Supervised Fine-Tuning (P-SFT) to enhance model anthropomorphism, though its generalization remains constrained by limited cross-task transferability.

Abstract

Reliable human-machine discrimination is becoming increasingly important as large language models and autonomous agents are deployed in online settings. Existing approaches evaluate whether a system can produce behavior or responses indistinguishable from those of a human, following the emphasis on outputs as a criterion for intelligence proposed by Alan Turing. Cognitive science offers an alternative perspective: evaluating the process by which behavior is produced. To test whether cognitive processes can reliably distinguish humans from machines, we introduce CogCAPTCHA30, a battery of 30 cognitive tasks designed to elicit diagnostic process-level features even when task performance is matched. Across the battery, process-level features provide stronger discriminative signal than performance metrics alone, reliably distinguishing humans from agents even under output matching (mean process-feature classifier AUC = 0.88). To evaluate agentic process differences, we compare off-the-shelf frontier agents (Claude Sonnet 4.5, GPT-5, Gemini 2.5 Pro), Centaur (a language model fine-tuned on 10.7M human decisions), and two task-specific fine-tuning approaches applied to Qwen2.5-1.5B-Instruct: action-level supervised fine-tuning (A-SFT) and process-level fine-tuning (P-SFT), which directly optimizes process features. Broad fine-tuning on human decisions improves human-like task processes relative to off-the-shelf agents, while task-specific process-level supervision further improves behavioral mimicry. However, this advantage diminishes under cross-task transfer when supervised process targets do not naturally generalize across tasks. Explicit process-level supervision can improve human behavioral mimicry, but only if appropriate task-specific process representations are available, highlighting process specification as a bottleneck for achieving human-like cognitive processes in machines.

Problem

• human-machine discrimination
• cognitive processes
• behavioral mimicry
• process-level features
• Turing test

Innovation

• process-level discrimination
• CogCAPTCHA30
• cognitive process modeling
• process-level fine-tuning
• human-machine distinction