What You See Is What You Get: Observation-Aligned Supervision for Chart-to-Code Generation

2026-07-06

AI Summary

This work addresses a critical limitation in existing chart-to-code generation methods, which rely on reference code containing unobservable latent variables for supervision, often leading to model hallucination and over-specification. The study systematically identifies this issue and introduces an observation-aligned supervision framework that restricts training objectives to quantities directly inferable from chart images—such as boxplot statistics, pie chart proportions, and histogram bin weights—ensuring alignment between supervision signals and visual observations. By integrating chart understanding from vision-language models with supervised fine-tuning and data rewriting techniques, the proposed approach significantly improves both the accuracy of observable attribute recovery and code executability on ChartMimic and ChartX benchmarks, demonstrating the pivotal role of observation-aligned supervision in enhancing model performance.

Abstract

Chart-to-code generation is commonly trained with supervised fine-tuning on reference plotting scripts, implicitly treating the gold code as a fully observable target. We argue that this assumption is often invalid: many chart programs contain latent raw variables that cannot be uniquely recovered from the rendered image. For example, a boxplot exposes summary statistics rather than original samples, a pie chart reveals proportions rather than arbitrary raw values, and a histogram shows bin-level mass rather than individual observations. Supervising models to reproduce such non-identifiable quantities encourages hallucination and over-specified code generation. We introduce Observation-Aligned supervision, a rewriting framework that replaces latent raw-data targets with quantities constrained by the visual observation: box statistics for boxplots, wedge percentages for pie charts, and bin weights for histograms. Applying this framework to chart-to-code training data from two sources, we obtain the Observation-Aligned supervision target data. Experiments across multiple VLMs on ChartMimic and ChartX demonstrate consistent improvements in observable value recovery, including under both-executable evaluation. Our results suggest that improving chart-to-code models requires not only more data or advanced learning objectives or algorithms, but also supervision targets that respect what is identifiable from the chart image.

Problem

• chart-to-code generation
• supervision
• latent variables
• visual observation
• identifiability

Innovation

• Observation-Aligned Supervision
• chart-to-code generation
• visual observability
• latent variable recovery
• executable evaluation