TraveLLaMA: Facilitating Multi-modal Large Language Models to Understand Urban Scenes and Provide Travel Assistance

2025-04-23

AI Summary

Existing general-purpose multimodal AI models lack domain-specific expertise and contextual understanding for urban tourism, hindering precise travel service delivery. To address this, we propose a multimodal large language model (MLLM) tailored to urban travel. We construct the first large-scale, domain-specific multimodal QA dataset for travel—comprising 220K samples, including 130K text-based Q&A pairs and 90K vision-based Q&A instances grounded in maps and real-world scenes—enabling fine-grained, context-aware multimodal reasoning and situational recommendation. Methodologically, our approach integrates architectures from LLaVA, Qwen-VL, and Shikra, leverages real-world travel forum data collection, employs GPT-augmented response generation, and applies domain-adaptive fine-tuning. Experiments demonstrate consistent improvements of 6.5–9.4% over generalist baselines on both text-only travel understanding and multimodal visual question answering, significantly enhancing capabilities in location parsing, business-hour querying, review summarization, and personalized itinerary planning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Tourism and travel planning increasingly rely on digital assistance, yet existing multimodal AI systems often lack specialized knowledge and contextual understanding of urban environments. We present TraveLLaMA, a specialized multimodal language model designed for urban scene understanding and travel assistance. Our work addresses the fundamental challenge of developing practical AI travel assistants through a novel large-scale dataset of 220k question-answer pairs. This comprehensive dataset uniquely combines 130k text QA pairs meticulously curated from authentic travel forums with GPT-enhanced responses, alongside 90k vision-language QA pairs specifically focused on map understanding and scene comprehension. Through extensive fine-tuning experiments on state-of-the-art vision-language models (LLaVA, Qwen-VL, Shikra), we demonstrate significant performance improvements ranging from 6.5%-9.4% in both pure text travel understanding and visual question answering tasks. Our model exhibits exceptional capabilities in providing contextual travel recommendations, interpreting map locations, and understanding place-specific imagery while offering practical information such as operating hours and visitor reviews. Comparative evaluations show TraveLLaMA significantly outperforms general-purpose models in travel-specific tasks, establishing a new benchmark for multi-modal travel assistance systems.

Problem

• Enhancing urban scene understanding for travel assistance
• Improving multimodal AI in travel-specific contextual tasks
• Bridging knowledge gaps in AI-based travel planning

Innovation

• Specialized multimodal model for urban travel
• Large-scale dataset with 220k QA pairs
• Fine-tuned vision-language models for travel tasks