CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoning

2026-05-19

AI Summary

This work addresses the inefficiency of traditional Chain-of-Thought (CoT) reasoning, which places deliberation before answer generation and often incurs redundant computation and latency when models can directly produce correct responses. The authors propose CopT, a novel “answer-first, think-later” paradigm that initially generates a draft answer and then employs a contrastive verification mechanism built upon continuous embeddings to dynamically assess answer reliability. By estimating uncertainty via reverse KL divergence, CopT enables adaptive reflection while preserving partial information from the initial response. Notably, this approach requires no additional training and achieves up to a 23% absolute accuracy gain on mathematical, programming, and agent-based reasoning tasks, while reducing token consumption by as much as 57% at comparable or higher accuracy levels.

Abstract

Chain-of-thought (CoT) is a standard approach for eliciting reasoning capabilities from large language models (LLMs). However, the common CoT paradigm treats thinking as a prerequisite for answering, which can delay access to plausible answers and incur unnecessary token costs even when the model is able to identify an answer before extended thinking, a behavior known as performative reasoning. In this paper, we introduce CopT, a reformulated reasoning pipeline that reverses the usual order of thinking and answering. Instead of thinking before answering, CopT first elicits a draft answer and then invokes subsequent on-policy thinking conditioned on its own draft answer for reflection and correction. To assess whether the draft answer should be trusted, CopT recasts continuous embeddings as inference-time contrastive verifiers. Specifically, it contrasts the model's support for the same generated tokens under discrete-token inputs and continuous-embedding inputs, yielding a sequence-level reverse KL estimator for answer reliability. Our analysis shows that under certain assumptions, the expected estimate equals the mutual information between the unresolved latent state and the emitted answer token, explaining why it captures answer-relevant uncertainty rather than arbitrary uncertainty in the latent state. When the answer is deemed insufficiently reliable, CopT performs further on-policy thinking, where a second KL estimator dynamically controls draft-answer visibility, preserving useful partial information while reducing the risk of being misled by unreliable content. Across mathematics, coding, and agentic reasoning tasks, CopT improves peak accuracy by up to 23% and reduces token usage by up to 57% at comparable or higher accuracy, without any additional training. The code is available at https://github.com/sdc17/CopT.

Problem

• Chain-of-thought
• performative reasoning
• answer reliability
• reasoning efficiency
• large language models

Innovation

• Contrastive Verification
• On-Policy Thinking
• Continuous Embeddings
• Reverse KL Estimator
• Chain-of-Thought Reasoning