Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking

2025-06-09

AI Summary

Small-scale large language models (LLMs) exhibit insufficient reasoning robustness under distributional shifts—such as numerical/nominal variable perturbations or insertion of distractor clauses. Method: We propose AbstraL, the first framework that formalizes abstract reasoning as a learnable reinforcement learning objective, overcoming the limitation of supervised fine-tuning in producing faithful abstract representations. AbstraL constructs a reward model from fine-grained abstract data and enforces abstraction consistency constraints to ensure compatibility between learned abstract representations and symbolic tools. Results: Experiments on the GSM perturbation benchmark demonstrate that AbstraL significantly mitigates performance degradation under distributional shifts. It substantively improves out-of-distribution reasoning generalization via learnable abstraction, validating its effectiveness in enhancing LLM robustness. AbstraL establishes a novel paradigm for robust reasoning in resource-constrained LLMs.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent studies have shown that large language models (LLMs), especially smaller ones, often lack robustness in their reasoning. I.e., they tend to experience performance drops when faced with distribution shifts, such as changes to numerical or nominal variables, or insertions of distracting clauses. A possible strategy to address this involves generating synthetic data to further"instantiate"reasoning problems on potential variations. In contrast, our approach focuses on"abstracting"reasoning problems. This not only helps counteract distribution shifts but also facilitates the connection to symbolic tools for deriving solutions. We find that this abstraction process is better acquired through reinforcement learning (RL) than just supervised fine-tuning, which often fails to produce faithful abstractions. Our method, AbstraL -- which promotes abstract reasoning in LLMs using RL on granular abstraction data -- significantly mitigates performance degradation on recent GSM perturbation benchmarks.

Problem

• Enhancing LLMs' robustness against reasoning distribution shifts
• Promoting abstract thinking to connect with symbolic tools
• Using reinforcement learning to improve faithful abstraction acquisition

Innovation

• Reinforces abstract thinking in LLMs
• Uses reinforcement learning for abstraction
• Mitigates performance drops on benchmarks