Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue

2026-06-19

AI Summary

As AI systems increasingly participate in online conversations, distinguishing human-authored from AI-generated multi-turn dialogues has become a critical challenge. This work proposes Inverse Turing Bench, a benchmark that formalizes the inverse Turing test as a quantifiable task by systematically evaluating language models’ ability to discriminate between human–AI and human–human dialogue pairs. The study reveals that statistical approaches suffer from semantic blind spots, while semantic methods are susceptible to role-based prompting, thereby advancing research on theory of mind in artificial intelligence. Experimental results show that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest performance, with accuracies of 89.41%, 77.92%, and 75.94%, respectively, highlighting both the current capabilities and limitations of state-of-the-art models on this task.

Abstract

As AI systems integrate into online spaces, differentiating them from humans in conversations is increasingly important. We present Inverse Turing Bench, a benchmark that evaluates LLMs and other models on their ability to differentiate humans and AI in multi-turn text. The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI. The task is to correctly identify which dialogue is human-only vs. human-AI. We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy: 89.41%, 77.92%, and 75.94% respectively. Our results suggest that statistical approaches to detection have semantic blind spots, but semantic approaches are susceptible to persona-prompting. Our work speaks to the Inverse Turing Test as a probe of LLM theory of mind, and motivates human-AI differentiation as a critical capability for AI systems. Our live benchmark can be found at https://huggingface.co/spaces/roc-hci/Inverse-Turing-Bench-Leaderboard (anonymity preserved).

Problem

• Inverse Turing Test
• human-AI differentiation
• dialogue evaluation
• language models as judges
• multi-turn conversation

Innovation

• Inverse Turing Test
• human-AI differentiation
• dialogue benchmark
• theory of mind
• persona-prompting