QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling

2024-09-21

arXiv.org

AI Summary

This work addresses the performance limitations of large language models (LLMs) on domain-specific multiple-choice question answering (MCQA) in telecommunications—where dense technical terminology, complex concepts, and stringent answer accuracy requirements impede robustness. We propose a collaborative optimization framework tailored for small language models (SLMs), integrating fine-tuning, retrieval-augmented generation (RAG), prompt engineering, and advanced reasoning strategies. Key innovations include a question-masked loss function and an option-shuffling mechanism to enhance discrimination among distractors. We instantiate the framework using Phi-2 and Falcon-7B, building a domain-adapted RAG system with dynamic option re-ranking and customized retrieval. Experiments demonstrate substantial gains: Falcon-7B achieves 49.3% accuracy (+24.6 percentage points), while Phi-2 reaches 84.65% (+42.6 percentage points), markedly surpassing prior SLM performance on professional MCQA tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.

Problem

• Enhance LLMs for telecommunications
• Improve multiple-choice question accuracy
• Use open-source smaller language models

Innovation

• Question-Masked loss technique
• Option Shuffling method
• Enhanced RAG framework