CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs

2026-07-06

AI Summary

This work addresses the tendency of large language models to generate infeasible plans in task planning due to insufficient awareness of intrinsic constraints. To mitigate this issue, the authors propose CARL (Constraint-Aware Reinforcement Learning), a novel framework that constructs a constraint-aware reward signal by comparing the model’s output distributions with and without explicit constraint inputs. This reward guides the model in an end-to-end manner to proactively attend to task constraints during planning. Notably, CARL operates without external solvers or auxiliary models and integrates seamlessly with mainstream reinforcement learning frameworks, offering strong scalability. Empirical evaluations on BlocksWorld, TravelPlanner, and T-Eval benchmarks demonstrate that CARL substantially outperforms standard reinforcement fine-tuning and state-of-the-art reasoning models, achieving significantly improved constraint adherence.

Abstract

Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications. This deficiency arises from a lack of systematic mechanisms to incorporate constraint information during the generation process. While existing approaches attempt to mitigate this by relying on external tools or task decomposition, they fail to enhance the model's intrinsic constraint awareness. To address this, we propose Constraint-Aware Reinforcement Learning (CARL), a novel RL framework designed to strengthen LLMs' intrinsic focus on constraints. CARL introduces a constraint-aware reward by comparing the model's output distributions under constrained and unconstrained inputs, encouraging constraint focus and penalizing neglect. Compatible with various RL frameworks and requiring no external solvers or top models, CARL enables scalable, end-to-end constraint-aware planning. Extensive experiments on BlocksWorld, TravelPlanner, and T-Eval demonstrate that CARL significantly outperforms standard Reinforcement Fine-Tuning (RFT) baselines and state-of-the-art reasoning models, exhibiting a markedly increased focus on constraints.

Problem

• constraint violation
• Large Language Models
• planning reliability
• constraint awareness

Innovation

• Constraint-Aware Reinforcement Learning
• Large Language Models
• Reward Design
• End-to-End Planning
• Constraint Satisfaction