DeepTravel: An End-to-End Agentic Reinforcement Learning Framework for Autonomous Travel Planning Agents

2025-09-26

AI Summary

Existing travel planning agents rely on handcrafted prompts and rigid pipelines, exhibiting limited autonomy and failing to establish a closed-loop of planning–execution–reflection in multi-step reasoning. This paper proposes an end-to-end agent reinforcement learning framework featuring (1) hierarchical reward modeling—supporting both trajectory-level consistency and episode-level correctness validation—and (2) response-augmented RL with failure experience replay. We construct a sandbox environment with toolized APIs for transportation, accommodation, and points of interest (POIs), enabling stable, efficient multi-step tool invocation and reflective optimization. Deployed in real-world Didi scenarios, our framework significantly enhances travel planning capabilities of compact models (e.g., Qwen3-32B), outperforming state-of-the-art large language models—including OpenAI o1/o3 and DeepSeek R1—on key operational metrics.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Multiagent Systems: Multiagent Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Travel planning (TP) agent has recently worked as an emerging building block to interact with external tools and resources for travel itinerary generation, ensuring enjoyable user experience. Despite its benefits, existing studies rely on hand craft prompt and fixed agent workflow, hindering more flexible and autonomous TP agent. This paper proposes DeepTravel, an end to end agentic reinforcement learning framework for building autonomous travel planning agent, capable of autonomously planning, executing tools, and reflecting on tool responses to explore, verify, and refine intermediate actions in multi step reasoning. To achieve this, we first construct a robust sandbox environment by caching transportation, accommodation and POI data, facilitating TP agent training without being constrained by real world APIs limitations (e.g., inconsistent outputs). Moreover, we develop a hierarchical reward modeling system, where a trajectory level verifier first checks spatiotemporal feasibility and filters unsatisfied travel itinerary, and then the turn level verifier further validate itinerary detail consistency with tool responses, enabling efficient and precise reward service. Finally, we propose the reply augmented reinforcement learning method that enables TP agent to periodically replay from a failures experience buffer, emerging notable agentic capacity. We deploy trained TP agent on DiDi Enterprise Solutions App and conduct comprehensive online and offline evaluations, demonstrating that DeepTravel enables small size LLMs (e.g., Qwen3 32B) to significantly outperform existing frontier LLMs such as OpenAI o1, o3 and DeepSeek R1 in travel planning tasks.

Problem

• Developing autonomous travel planning agents with flexible workflows
• Creating robust training environments without real API limitations
• Enhancing small LLMs to outperform frontier models in planning

Innovation

• End-to-end reinforcement learning framework for autonomous travel planning
• Hierarchical reward modeling system for efficient itinerary validation
• Reply-augmented reinforcement learning with failure experience replay