ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning

2025-02-07

AI Summary

Large language models (LLMs) often exhibit ambiguous reasoning and insufficient knowledge retrieval in multiple-choice question answering. Method: This paper proposes the zero-shot ARR prompting framework, which explicitly decomposes the task into three sequential stages: question intent analysis, relevant knowledge retrieval, and stepwise reasoning. It is the first to structurally integrate intent recognition, retrieval guidance, and chain-of-thought reasoning under a zero-shot setting—without fine-tuning or exemplars. Contribution/Results: Intent analysis is empirically validated as the key driver of performance gains. ARR demonstrates strong generalizability across model families (LLaMA, Qwen, Claude), architectures, and decoding strategies. On challenging multiple-choice QA benchmarks, it significantly outperforms zero-shot chain-of-thought and standard baselines. Ablation studies confirm the individual contributions of each component, while case analyses highlight its interpretability and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) achieve remarkable performance on challenging benchmarks that are often structured as multiple-choice question-answering (QA) tasks. Zero-shot Chain-of-Thought (CoT) prompting enhances reasoning in LLMs but provides only vague and generic guidance ("think step by step"). This paper introduces ARR, an intuitive and effective zero-shot prompting method that explicitly incorporates three key steps in QA solving: analyzing the intent of the question, retrieving relevant information, and reasoning step by step. Comprehensive experiments across diverse and challenging QA tasks demonstrate that ARR consistently improves the Baseline (without ARR prompting) and outperforms CoT. Ablation and case studies further validate the positive contributions of each component: analyzing, retrieving, and reasoning. Notably, intent analysis plays a vital role in ARR. Additionally, extensive evaluations across various model sizes, LLM series, and generation settings solidify the effectiveness, robustness, and generalizability of ARR.

Problem

• Enhances QA with LLMs
• Improves zero-shot prompting
• Analyzes, retrieves, reasons effectively

Innovation

• Analyzes question intent
• Retrieves relevant information
• Reasons step-by-step