LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling

2026-05-13

AI Summary

Although large language models possess intrinsic metacognitive signals, they struggle to actively leverage them during reasoning. Inspired by the Nelson–Narens metacognitive framework, this work proposes the first test-time metacognitive control architecture that explicitly converts pre-answer "feeling-of-knowing" (FOK) and post-answer "judgment-of-learning" (JOL) into dynamic control signals to decide whether to trust, retry, or aggregate responses—without requiring parameter updates or task-specific fine-tuning. Experiments with Claude Sonnet-4.6 demonstrate substantial performance gains across textual, code, and multimodal benchmarks, raising overall accuracy from 48.3% to 56.9% and establishing new state-of-the-art results on HLE-Verified, LiveCodeBench v6, and R-Bench-V.

Abstract

Large language models (LLMs) often expose useful signals of self-monitoring: before solving a problem, they can estimate whether they are likely to succeed, and after solving it, they can judge whether their answer is likely to be correct. However, these signals are typically measured or elicited in isolation, rather than used to control inference. In this work, we ask whether LLMs possess latent metacognitive ability that can be turned into effective test-time control. Inspired by the Nelson--Narens theory from cognitive psychology, we propose a metacognitive harness that separates monitoring from reasoning. For each problem, the model first reports a pre-solve feeling-of-knowing (FOK) signal; after each solve attempt, it reports a post-solve judgment-of-learning (JOL) signal. Rather than treating these signals as passive confidence estimates, the harness turns them into an explicit control interface for reasoning: it decides when to trust the current solution, when to retry with compact metacognitive feedback, and when to pass multiple attempts to a final aggregator. Across text, code, and multimodal reasoning benchmarks, our harness substantially improves a fixed Claude Sonnet-4.6 base model without parameter updates or benchmark-specific fine-tuning. On the evaluated public benchmark snapshots, it raises pooled accuracy from 48.3 to 56.9 and exceeds the strongest listed leaderboard entries on the three primary evaluation settings: HLE-Verified, LiveCodeBench v6, and R-Bench-V. These results suggest that strong LLMs may already possess useful metacognitive ability, but require an explicit control harness to act on it during reasoning.

Problem

• metacognition
• large language models
• self-monitoring
• test-time scaling
• reasoning control

Innovation

• metacognitive harness
• feeling-of-knowing
• judgment-of-learning
• test-time scaling
• self-monitoring