Why Large Language Models Fail at Tabular Prediction
This study investigates the fundamental reasons behind the poor performance of large language models (LLMs) on tabular prediction tasks under zero-shot settings without fine-tuning or external tools. By systematically evaluating five hypotheses across 31 benchmark datasets, integrating random linear projections with grid consistency analysis, and comparing against 252 classical model configurations, the work identifies dimensionality growth as the primary factor driving significant LLM performance degradation. The findings reveal that while LLMs exhibit behavior akin to local distance-based methods in low-dimensional spaces (e.g., two dimensions), they display a distinct and irreproducible degeneration pattern in high-dimensional settings—one that cannot be emulated by any classical model. In contrast, traditional methods maintain stable or even improved performance as dimensionality increases.