Cohort-based Semantic Labeling: AI-Enabled Recovery of Visualization Semantics from Deployed SVGs

2026-06-08

AI Summary

Existing post-deployment SVG visualizations lack high-level semantic structure, hindering downstream tasks such as querying, accessibility, interpretation, personalization, and transformation. To address this limitation, this work proposes the CSL multi-stage pipeline, which reduces the semantic search space through group decomposition and integrates AI-driven clustering, hybrid semantic reasoning, and deterministic structural validation to achieve context-sensitive yet structurally robust semantic recovery, yielding annotated Semantic SVGs (SSVGs). Evaluated on 102 real-world SVGs, the method achieves macro-averaged accuracies of 0.822, 0.853, and 0.860 for tag type, visualization role, and data role, respectively. The grouping strategy significantly enhances performance (p < 0.001), and repeated annotations across 100 trials demonstrate consistency exceeding 91.9%.

Abstract

Many web-based visualizations are deployed as Scalable Vector Graphics (SVG), a format that faithfully preserves visual appearance but typically omits the higher-level semantic structure needed for machine interpretation. Once rendered and published, information about a visualization's components, roles, and encodings is no longer explicitly available, limiting downstream operations such as querying, accessibility augmentation, explanation, personalization, and transformation. To address this gap, we introduce CSL, an AI-enabled, multi-stage pipeline for automatically recovering visualization semantics from deployed SVGs through two complementary mechanisms: (1) cohort-based decomposition, which organizes heterogeneous SVG primitives into structurally coherent subsets that reduce the semantic assignment space, and (2) hybrid semantic grounding, which combines model-based inference with deterministic structural validation and propagation to make labeling both context-sensitive and structurally anchored. CSL produces Semantic SVG (SSVG), a representation in which SVG elements are annotated with graphical mark type, visualization role, and data role. We implemented CSL as an end-to-end prototype and evaluated it on 102 SVG visualizations, achieving global macro-averaged accuracies of 0.822 for mark type, 0.853 for visualization role, and 0.860 for data-role recovery. An ablation against a non-cohort whole-chart baseline showed that cohorting significantly improves accuracy (paired t-test: t > 20, p < 0.001; Cohen's d > 2.0), and repeated labeling of a randomly selected SVG over 100 runs yielded mean agreement above 91.9% across all three attributes. These results provide strong evidence that CSL can transform deployed SVGs into machine-usable semantic representations, enabling more accessible, adaptive, and user-steerable visualization systems.

Problem

• semantic labeling
• SVG
• visualization semantics
• machine interpretation
• cohort-based decomposition

Innovation

• cohort-based decomposition
• semantic labeling
• SVG recovery
• hybrid semantic grounding
• visualization semantics