🤖 AI Summary
This study addresses the challenge of automated customer requirement analysis in the tourism industry. We propose a large language model (LLM)-based framework for zero-shot requirement extraction and comparative evaluation from user-generated texts. Methodologically, we systematically benchmark open-source (Llama-3, Mistral 7B) and proprietary (GPT-4, Gemini) LLMs on TripAdvisor and Reddit data, evaluating their performance in unsupervised requirement identification and structured summarization via prompt engineering and multi-dimensional automatic metrics (BERTScore, ROUGE, BLEU). Our key contributions are twofold: (1) the first empirical demonstration in the tourism domain that compact open-source models—specifically Mistral 7B—achieve 96% of GPT-4’s requirement identification accuracy while reducing inference cost by 83%; and (2) a principled LLM selection framework balancing performance, cost-efficiency, and domain customizability, validating the practical viability and cost-effectiveness of lightweight open-source models in real-world tourism applications.
📝 Abstract
In the rapidly evolving landscape of Natural Language Processing (NLP), Large Language Models (LLMs) have emerged as powerful tools for many tasks, such as extracting valuable insights from vast amounts of textual data. In this study, we conduct a comparative analysis of LLMs for the extraction of travel customer needs from TripAdvisor and Reddit posts. Leveraging a diverse range of models, including both open-source and proprietary ones such as GPT-4 and Gemini, we aim to elucidate their strengths and weaknesses in this specialized domain. Through an evaluation process involving metrics such as BERTScore, ROUGE, and BLEU, we assess the performance of each model in accurately identifying and summarizing customer needs. Our findings highlight the efficacy of opensource LLMs, particularly Mistral 7B, in achieving comparable performance to larger closed models while offering affordability and customization benefits. Additionally, we underscore the importance of considering factors such as model size, resource requirements, and performance metrics when selecting the most suitable LLM for customer needs analysis tasks. Overall, this study contributes valuable insights for businesses seeking to leverage advanced NLP techniques to enhance customer experience and drive operational efficiency in the travel industry.