Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

2026-03-25

AI Summary

This work addresses the challenges faced by multimodal large language models in table–vision fusion tasks—namely, structural variability, missing data, implicit dependencies, and task heterogeneity—by proposing the Thinking with Tables (TWT) framework, which introduces neuro-symbolic reasoning to this domain for the first time. TWT leverages program-aided code generation and interaction with an external environment to perform information extraction and element modeling, enabling deep comprehension of complex table–vision data. Evaluated across eight benchmark datasets, the proposed method achieves an average accuracy improvement of 10% over existing baselines, matching or even surpassing the performance of state-of-the-art commercial large language models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10\% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks. Models and codes are available at https://github.com/kunyang-YU/Thinking-with-Tables

Problem

• Tabular-Vision Multi-Modal Understanding
• structural variability
• data incompleteness
• feature dependencies
• pipeline heterogeneity

Innovation

• neuro-symbolic reasoning
• tabular-vision understanding
• program-aided reasoning
• multimodal LLMs
• code-based interaction