Playing Psychic: Using Thought Trees to Predict Reasoning Models Accuracy on Coding Tasks

2026-04-18

AI Summary

Existing reasoning models lack systematic evaluation on real-world programming tasks, as prevailing benchmarks are largely confined to competitive programming problems and fail to comprehensively assess reasoning capabilities. This work proposes an evaluation framework that programmatically generates programming tasks of arbitrary difficulty and structure, along with a structured Tree-of-Thought representation to capture model reasoning trajectories. We find that structural features of the reasoning tree strongly predict trajectory correctness, enabling the construction of a lightweight classifier that automatically identifies and triggers retries for anomalous trajectories. This approach significantly improves model performance on low-complexity tasks and establishes a novel paradigm for predicting correctness based on reasoning structure.

Technology Category

• Reasoning under Uncertainty: Probabilistic Programming
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Planning, Routing, and Scheduling: Model-Based Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advances in large language models (LLMs) have shown that test-time scaling can substantially improve model performance on complex tasks, particularly in the coding domain. Under this paradigm, models use a larger token budget during inference to generate intermediate reasoning traces before producing a final answer. However, current evaluations primarily rely on competitive programming benchmarks, which may not capture the full range of reasoning abilities. In this work, we perform a systematic study of frontier reasoning models to understand their performance on real-world coding benchmarks. To gain more insights into the performance of such models, we devise a programmatic way to {\em automatically generate} coding tasks of arbitrary difficulty and structure from existing benchmarks. Using this framework, our analysis reveals that the structure of a reasoning trace, not just its contents, is a strong predictor of correctness. Motivated by this, we propose structured thought-trees as means to represent reasoning traces. To illustrate their use, we train a lightweight classifier on features extracted from thought-trees to predict trace correctness, and demonstrate that flagging and retrying structurally anomalous traces based on the extracted features yields consistent gains at lower complexity levels.

Problem

• reasoning models
• coding tasks
• reasoning traces
• thought trees
• model accuracy

Innovation

• thought trees
• reasoning traces
• test-time scaling
• code generation
• structured prediction