General Reasoning Requires Learning to Reason from the Get-go

2025-02-26

AI Summary

Large language models (LLMs) exhibit fragile cross-context algorithmic generalization due to tight coupling between reasoning and knowledge, undermining robustness and transferability. Method: We propose a novel “reasoning–knowledge decoupling” paradigm: (1) replacing standard next-token prediction with from-scratch reinforcement learning pretraining; (2) designing a synthetic algorithmic task curriculum to explicitly induce transferable reasoning priors; and (3) integrating small-context modeling with retrieval-augmented architecture to suppress token-level spurious correlations and disentangle local perception from global reasoning. Results: Our approach significantly improves cross-context generalization on heterogeneous programming-language algorithm tasks and successfully transfers to natural language inference. It constitutes the first systematic framework for building AGI-level transferable reasoning systems, demonstrating principled decoupling of reasoning mechanisms from domain-specific knowledge representations.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have demonstrated impressive real-world utility, exemplifying artificial useful intelligence (AUI). However, their ability to reason adaptively and robustly -- the hallmarks of artificial general intelligence (AGI) -- remains fragile. While LLMs seemingly succeed in commonsense reasoning, programming, and mathematics, they struggle to generalize algorithmic understanding across novel contexts. Our experiments with algorithmic tasks in esoteric programming languages reveal that LLM's reasoning overfits to the training data and is limited in its transferability. We hypothesize that the core issue underlying such limited transferability is the coupling of reasoning and knowledge in LLMs. To transition from AUI to AGI, we propose disentangling knowledge and reasoning through three key directions: (1) pretaining to reason using RL from scratch as an alternative to the widely used next-token prediction pretraining, (2) using a curriculum of synthetic tasks to ease the learning of a extit{reasoning prior} for RL that can then be transferred to natural language tasks, and (3) learning more generalizable reasoning functions using a small context window to reduce exploiting spurious correlations between tokens. Such a reasoning system coupled with a trained retrieval system and a large external memory bank as a knowledge store can overcome several limitations of existing architectures at learning to reason in novel scenarios.

Problem

• LLMs struggle with adaptive, robust reasoning.
• Limited transferability of algorithmic understanding in LLMs.
• Need to disentangle knowledge and reasoning in LLMs.

Innovation

• RL from scratch pretraining
• Curriculum of synthetic tasks
• Small context window reasoning