IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarization

2024-07-15

arXiv.org

AI Summary

To address the challenge in query-focused summarization (QFS) where large language models struggle to jointly model long documents and achieve fine-grained query alignment, this paper proposes the Query-aware HyperExpert framework and the Query-focused Infini-attention mechanism. The former employs query-aware hyper-expert routing for modular semantic adaptation, while the latter integrates query-driven sparse attention constraints into infinite-context modeling. By synergistically combining extractive summarization paradigms with the strong generative capabilities of LLMs, our approach achieves significant improvements over state-of-the-art methods across multiple standard QFS benchmarks. It delivers high accuracy, strong generalization across diverse query types and document domains, and improved inference efficiency. The implementation is publicly available.

Technology Category

• Natural Language Processing: Summarization
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization. With the advent of large language models (LLMs), shows their impressive capability of textual understanding through large-scale pretraining, which implies the great potential of extractive snippet generation. In this paper, we systematically investigated two indispensable characteristics that the LLMs-based QFS models should be harnessed, Lengthy Document Summarization and Efficiently Fine-grained Query-LLM Alignment, respectively. Correspondingly, we propose two modules called Query-aware HyperExpert and Query-focused Infini-attention to access the aforementioned characteristics. These innovations pave the way for broader application and accessibility in the field of QFS technology. Extensive experiments conducted on existing QFS benchmarks indicate the effectiveness and generalizability of the proposed approach. Our code is publicly available at https://github.com/DCDmllm/IDEAL_Summary.

Problem

• Large Language Models
• Query-Focused Summarization
• Information Retrieval

Innovation

• Query-Focused Summarization
• Long Document Processing
• Precision Query Matching