Reading with Intent

2024-08-20

arXiv.org

AI Summary

Real-world RAG systems frequently retrieve stylistically diverse, open-web texts, whereas existing benchmarks (e.g., Wikipedia corpora) exhibit neutral tone and limited pragmatic diversity, hindering detection of implicit semantics such as irony. To address this, we propose an irony-augmented retrieval–reading collaborative framework—the first to systematically integrate irony awareness. Our method synthesizes controllable irony-rich training data, employs intent-guided prompting to explicitly model speaker intent, and jointly optimizes both retriever and reader modules. Ablation studies confirm the efficacy of each component. Evaluated on the Natural Questions benchmark, our approach significantly improves irony identification accuracy while boosting retrieval relevance (+4.2%) and answer faithfulness (+5.8%). This work establishes a novel paradigm for enhancing RAG robustness in authentic, pragmatically rich contexts.

Technology Category

• Natural Language Processing: Question Answering
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Common-Sense Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Retrieval augmented generation (RAG) systems augment how knowledge language models are by integrating external information sources such as Wikipedia, internal documents, scientific papers, or the open internet. RAG systems that rely on the open internet as their knowledge source have to contend with the complexities of human-generated content. Human communication extends much deeper than just the words rendered as text. Intent, tonality, and connotation can all change the meaning of what is being conveyed. Recent real-world deployments of RAG systems have shown some difficulty in understanding these nuances of human communication. One significant challenge for these systems lies in processing sarcasm. Though the Large Language Models (LLMs) that make up the backbone of these RAG systems are able to detect sarcasm, they currently do not always use these detections for the subsequent processing of text. To address these issues, in this paper, we synthetically generate sarcastic passages from Natural Question's Wikipedia retrieval corpus. We then test the impact of these passages on the performance of both the retriever and reader portion of the RAG pipeline. We introduce a prompting system designed to enhance the model's ability to interpret and generate responses in the presence of sarcasm, thus improving overall system performance. Finally, we conduct ablation studies to validate the effectiveness of our approach, demonstrating improvements in handling sarcastic content within RAG systems.

Problem

• Handling diverse emotional tones in RAG-retrieved internet text
• Adapting neutral text to varied emotional and sarcastic styles
• Improving LLMs' interpretation of emotionally complex retrieved content

Innovation

• Dataset with emotionally inflected and sarcastic text
• Emotion translation model for tone adaptation
• Prompt-based method for better text interpretation