On the Robustness of Language Models for Tabular Question Answering

2024-06-18

arXiv.org

AI Summary

This work systematically evaluates the robustness of large language models (LLMs) on table question answering (TQA), examining the effects of in-context learning, model scale, instruction tuning, and domain shift. Experiments span three benchmark datasets—WTQ, TAT-QA, and SciTab—and involve structured perturbations (format, semantic, and noise) alongside data augmentation. Key findings: (1) instruction tuning yields an average +12.7% accuracy gain and substantially improves robustness; (2) newer models (e.g., GPT-4, Claude series) outperform earlier generations; (3) WTQ suffers from severe data contamination, inflating performance estimates, whereas TAT-QA and SciTab better reflect true generalization capability. The study further advocates for structural-aware self-attention mechanisms and domain-adaptive modeling to enhance reliability and trustworthiness in table understanding.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Large Language Models (LLMs), already shown to ace various text comprehension tasks have also remarkably been shown to tackle table comprehension tasks without specific training. While previous research has explored LLM capabilities with tabular dataset tasks, our study assesses the influence of extit{in-context learning}, extit{model scale}, extit{instruction tuning}, and extit{domain biases} on Tabular Question Answering (TQA). We evaluate the robustness of LLMs on Wikipedia-based extbf{WTQ}, financial report-based extbf{TAT-QA}, and scientific claims-based extbf{SCITAB}, TQA datasets, focusing on their ability to interpret tabular data under various augmentations and perturbations robustly. Our findings indicate that instructions significantly enhance performance, with recent models exhibiting greater robustness over earlier versions. However, data contamination and practical reliability issues persist, especially with extbf{WTQ}. We highlight the need for improved methodologies, including structure-aware self-attention mechanisms and better handling of domain-specific tabular data, to develop more reliable LLMs for table comprehension.

Problem

• Assessing LLM robustness in Tabular Question Answering under various factors
• Evaluating LLM performance on diverse TQA datasets with perturbations
• Identifying limitations like data contamination and domain-specific reliability issues

Innovation

• In-context learning enhances TQA performance
• Structure-aware self-attention improves table comprehension
• Domain-specific data handling boosts reliability