Seek and Solve Reasoning for Table Question Answering

2024-09-09

arXiv.org

AI Summary

Large language models (LLMs) struggle with complex table-based question answering (TQA), facing challenges in accurately locating relevant cells/relations and performing reliable logical reasoning. To address this, we propose the Seek-and-Solve reasoning paradigm: first precisely identifying salient cells and their semantic relations, then performing structured computation grounded in these locational cues—unified into a single SS-CoT (Seek-and-Solve Chain-of-Thought) reasoning chain. We empirically demonstrate that intermediate reasoning steps during task simplification are pedagogically more valuable than final simplified outputs, motivating a distillable one-step prompting scheme and an In-Context Learning demonstration distillation strategy guided by SS-CoT paths. Evaluated across multiple TQA benchmarks, our approach significantly improves accuracy and robustness while maintaining inference efficiency and cross-table generalization, validating that explicit stepwise reasoning effectively unlocks LLMs’ deeper reasoning capabilities.

Technology Category

• Natural Language Processing: Question Answering
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The complexities of table structures and question logic make table-based question answering (TQA) tasks challenging for Large Language Models (LLMs), often requiring task simplification before solving. This paper reveals that the reasoning process during task simplification may be more valuable than the simplified tasks themselves and aims to improve TQA performance by leveraging LLMs' reasoning capabilities. We propose a Seek-and-Solve pipeline that instructs the LLM to first seek relevant information and then answer questions, integrating these two stages at the reasoning level into a coherent Seek-and-Solve Chain of Thought (SS-CoT). Additionally, we distill a single-step TQA-solving prompt from this pipeline, using demonstrations with SS-CoT paths to guide the LLM in solving complex TQA tasks under In-Context Learning settings. Our experiments show that our approaches result in improved performance and reliability while being efficient. Our findings emphasize the importance of eliciting LLMs' reasoning capabilities to handle complex TQA tasks effectively.

Problem

• Large Language Models
• Complex Tabular Data
• Accuracy and Reliability

Innovation

• Seek-and-Solve CoT
• Complex Problem Solving
• Large Language Models