The Power of Order: Fooling LLMs with Adversarial Table Permutations

2026-05-01

AI Summary

This work reveals that large language models (LLMs) are highly sensitive to row and column ordering in table-based question answering, where semantically equivalent permutations can substantially degrade performance. The study introduces Adversarial Table Permutation (ATP), a gradient-based optimization method that efficiently generates table reorderings which preserve semantic content yet significantly impair model accuracy. Experimental results demonstrate that ATP consistently reduces performance across multiple state-of-the-art LLMs, exposing a pervasive vulnerability in their handling of structured data. This finding underscores the need for improved robustness mechanisms when processing tabular information and opens new avenues for enhancing model reliability in real-world applications involving structured inputs.

Abstract

Large Language Models have achieved remarkable success and are increasingly deployed in critical applications involving tabular data, such as Table Question Answering. However, their robustness to the structure of this input remains a critical, unaddressed question. This paper demonstrates that modern LLMs exhibit a significant vulnerability to the layout of tabular data. Specifically, we show that semantically-invariant permutations of rows and columns - rearrangements that do not alter the table's underlying information - are sometimes sufficient to cause incorrect or inconsistent model outputs. To systematically probe this vulnerability, we introduce Adversarial Table Permutation, a novel, gradient-based attack that efficiently identifies worst-case permutations designed to maximally disrupt model performance. Our extensive experiments demonstrate that ATP significantly degrades the performance of a wide range of LLMs. This reveals a pervasive vulnerability across different model sizes and architectures, including the most recent and popular models. Our findings expose a fundamental weakness in how current LLMs process structured data, underscoring the urgent need to develop permutation-robust models for reliable, real-world applications.

Problem

• Large Language Models
• Tabular Data
• Adversarial Permutation
• Robustness
• Table Question Answering

Innovation

• Adversarial Table Permutation
• Large Language Models
• Tabular Data Robustness
• Semantic-Invariant Permutation
• Gradient-Based Attack