Passage Query Methods for Retrieval and Reranking in Conversational Agents

2025-02-28

AI Summary

Existing conversational information retrieval (IR) methods struggle with cross-domain context understanding and document matching in multi-turn dialogues. Method: We propose Paragraph-aligned Query Generation (PQs), the first paragraph-level query modeling paradigm, which integrates semantic alignment of paragraph queries, multi-granularity paragraph encoding, and weighted re-ranking. PQs employs dual strategies—short and long paragraphs—and operates efficiently on a lightweight Llama model within a generate-retrieve-generate (GRG) pipeline. Contribution/Results: Evaluated on TREC iKAT, PQs significantly improves retrieval accuracy and response consistency. The Llama-based system achieves top-ranked performance—comparable to GPT-4 and surpassing the official baseline—demonstrating the effectiveness and generalizability of paragraph-level alignment for conversational IR.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

This paper presents our approach to the TREC Interactive Knowledge Assistance Track (iKAT), which focuses on improving conversational information-seeking (CIS) systems. While recent advancements in CIS have improved conversational agents' ability to assist users, significant challenges remain in understanding context and retrieving relevant documents across domains and dialogue turns. To address these issues, we extend the Generate-Retrieve-Generate pipeline by developing passage queries (PQs) that align with the target document's expected format to improve query-document matching during retrieval. We propose two variations of this approach: Weighted Reranking and Short and Long Passages. Each method leverages a Meta Llama model for context understanding and generating queries and responses. Passage ranking evaluation results show that the Short and Long Passages approach outperformed the organizers' baselines, performed best among Llama-based systems in the track, and achieved results comparable to GPT-4-based systems. These results indicate that the method effectively balances efficiency and performance. Findings suggest that PQs improve semantic alignment with target documents and demonstrate their potential to improve multi-turn dialogue systems.

Problem

• Improving conversational information-seeking systems' context understanding.
• Enhancing query-document matching across domains and dialogue turns.
• Balancing efficiency and performance in multi-turn dialogue systems.

Innovation

• Developed passage queries for better document matching
• Used Meta Llama for context understanding and generation
• Introduced Short and Long Passages for improved ranking