To Think or Not to Think: The Hidden Cost of Meta-Training with Excessive CoT Examples

2025-12-04

AI Summary

Large language models (LLMs) suffer degraded few-shot reasoning transfer performance during meta-training when over-reliant on chain-of-thought (CoT) examples. Method: We propose CoT-Recipe, a method that dynamically modulates the ratio of CoT to non-CoT samples during meta-training to optimize sequence-level sample composition. Contribution/Results: Experiments reveal that “excessive CoT” severely impairs generalization under CoT supervision scarcity—validating the “less-is-more” principle in meta-training. Implemented within the CoT-ICL Lab framework, CoT-Recipe is evaluated on symbolic reasoning tasks using mainstream LLMs (e.g., Qwen2.5). Without requiring in-context CoT demonstrations, it achieves up to a 300% accuracy gain and an overall performance improvement of 130%, significantly enhancing cross-task reasoning robustness and generalization under low-supervision regimes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Metareasoning and Metaheuristics
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Chain-of-thought (CoT) prompting combined with few-shot in-context learning (ICL) has unlocked significant reasoning capabilities in large language models (LLMs). However, ICL with CoT examples is ineffective on novel tasks when the pre-training knowledge is insufficient. We study this problem in a controlled setting using the CoT-ICL Lab framework, and propose meta-training techniques to learn novel abstract reasoning tasks in-context. Although CoT examples facilitate reasoning, we noticed that their excessive inclusion during meta-training degrades performance when CoT supervision is limited. To mitigate such behavior, we propose CoT-Recipe, a formal approach to modulate the mix of CoT and non-CoT examples in meta-training sequences. We demonstrate that careful modulation via CoT-Recipe can increase the accuracy of transformers on novel tasks by up to 300% even when there are no CoT examples available in-context. We confirm the broader effectiveness of these techniques by applying them to pretrained LLMs (Qwen2.5 series) for symbolic reasoning tasks and observing gains of up to 130% in accuracy.

Problem

• Excessive CoT examples degrade meta-training performance
• CoT-Recipe modulates CoT and non-CoT example mix
• Improves accuracy on novel tasks without in-context CoT

Innovation

• CoT-Recipe modulates CoT and non-CoT examples in meta-training
• Meta-training techniques improve novel abstract reasoning tasks in-context
• Careful modulation increases transformer accuracy up to 300% on novel tasks