Available but Unclaimed: An Empirical Study of Human-AI Synergy

2026-09-15

AI Summary

研究通过实验探讨了人类与AI协作解题时的表现，发现辅助准确性随AI能力增加而提高，但不同任务和模型间存在差异。

Abstract

People increasingly reason with large language models (LLMs), yet complementary capabilities do not guarantee outperforming both components. In a between-subjects study, participants (N=535) solved a 40-item battery of matrix reasoning, mental rotation, syllogisms, and letter-string analogies, unaided or with GPT-5.6-Luna, Claude Opus 4.8, Gemini 3.6 Flash, or Kimi K3. Each assisted trial required consultation with the model. Each model answered every item alone 100 times under matched elicitation. The assisted-unaided accuracy difference increased with item-level LLM competence. Deference varied across tasks and increased with competence within tasks. Post-advice confidence distinguished correct from incorrect answers less strongly than unaided confidence. In a reference comparison, about half the increase in LLM accuracy carried through to assisted accuracy. How much of that accuracy gain reached participants differed across the models. These findings motivate evaluating LLMs in interaction with humans and designing support for selective deference that preserves independent reasoning.

Problem

• Human-AI Synergy
• Large Language Models
• Reasoning Tasks
• Deference
• Accuracy

Innovation

• Human-AI Synergy
• Selective Deference
• Confidence Post-Advice