Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models

2025-09-22

AI Summary

To address the challenge of generating high-fidelity LaTeX code from complex table images—characterized by large dimensions, deep nesting, rich semantics, or irregular content—this paper proposes an end-to-end dual-reward reinforcement learning framework. The method leverages a multimodal large language model and jointly optimizes for two complementary rewards: LaTeX structural correctness and visual fidelity of the rendered output. Policy optimization is performed via Group Relative Policy Optimization (GRPO), and the model is fine-tuned on a large-scale table–LaTeX paired dataset. Evaluation employs a hybrid metric combining TEDS-Structure and CW-SSIM. Experiments demonstrate state-of-the-art performance across multiple benchmarks, with particularly pronounced gains on structurally complex tables. The approach achieves high accuracy, strong robustness to diverse table layouts, and publication-ready output quality.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In this work, we address the task of table image to LaTeX code generation, with the goal of automating the reconstruction of high-quality, publication-ready tables from visual inputs. A central challenge of this task lies in accurately handling complex tables -- those with large sizes, deeply nested structures, and semantically rich or irregular cell content -- where existing methods often fail. We begin with a comprehensive analysis, identifying key challenges and highlighting the limitations of current evaluation protocols. To overcome these issues, we propose a reinforced multimodal large language model (MLLM) framework, where a pre-trained MLLM is fine-tuned on a large-scale table-to-LaTeX dataset. To further improve generation quality, we introduce a dual-reward reinforcement learning strategy based on Group Relative Policy Optimization (GRPO). Unlike standard approaches that optimize purely over text outputs, our method incorporates both a structure-level reward on LaTeX code and a visual fidelity reward computed from rendered outputs, enabling direct optimization of the visual output quality. We adopt a hybrid evaluation protocol combining TEDS-Structure and CW-SSIM, and show that our method achieves state-of-the-art performance, particularly on structurally complex tables, demonstrating the effectiveness and robustness of our approach.

Problem

• Automating reconstruction of publication-ready tables from visual inputs
• Accurately handling complex tables with nested structures and irregular content
• Improving visual output quality through dual-reward reinforcement learning

Innovation

• Reinforced multimodal language model fine-tuned on table data
• Dual-reward reinforcement learning with Group Relative Policy Optimization
• Combines structure-level and visual fidelity rewards for optimization