Evaluation Ability Does Not Imply Optimization Utility: LLM-as-a-Judge Signals in Closed-Loop Table Recognition

2026-07-14

AI Summary

This study investigates the effectiveness of large language models as judges (LLM-as-a-judge) in providing optimization signals for closed-loop table recognition. Leveraging the FinTabNet and OmniDocBench datasets, and combining deterministic TEDS evaluation with structure-preserving instruction constraints, a 2×2 controlled experiment reveals that LLM-provided judgment signals are weak and unreliable—random selection often outperforms score-based iterative strategies. Although structure-preserving constraints substantially reduce severe structural errors, they fail to improve overall performance. The work is the first to demonstrate a critical disconnect between an LLM’s evaluative capability and its utility in optimization, thereby challenging the validity of relying solely on judgment scores for iterative refinement and underscoring the necessity of incorporating stronger verification signals to achieve effective closed-loop optimization.

Abstract

LLM-as-a-judge is widely used to provide feedback and selection signals in closedloop regeneration, but this use remains insufficiently validated. We study it in table recognition, where deterministic TEDS evaluation provides a controlled testbed, using FinTabNet and OmniDocBench. Three findings emerge. First, judge signals were weak on both datasets: scores frequently tied, rankings were not reproducible, and the only selection policy that beat random on both datasets depended on an earliest-iteration tie rule, so its advantage cannot be attributed to the judge scores alone. Iteration produced better candidates, but the judge failed to recover them. Second, severe losses occurred even without specific judge feedback. A structurepreserving instruction significantly reduced the severe-loss rate on FinTabNet and was directionally consistent on OmniDocBench. The contrasts support target-preservation failure under unconstrained regeneration as a proximate mechanism of the observed severe losses. Third, the structure-preservation constraint reduced the severe-loss tail but produced no improvement. In an exploratory 2x2 analysis, the same protection was not stably observed when judge feedback was retained. These results do not dispute the value of LLMs as evaluators. Instead, they show that evaluation ability does not imply optimization utility. Iterative refinement requires, at minimum, a verification signal that deterministically detects structural change, rather than judge scores alone.

Problem

• LLM-as-a-judge
• table recognition
• closed-loop regeneration
• optimization utility
• evaluation ability

Innovation

• LLM-as-a-judge
• closed-loop regeneration
• table recognition
• structural preservation
• optimization utility