Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing

2026-07-02

AI Summary

This work addresses the limitations of existing pixel-level visual regression testing (VRT) for web user interfaces, which struggles to distinguish rendering noise from genuine defects, leading to high false-positive rates and lacking public benchmarks that support natural language descriptions of UI changes. To bridge this gap, we introduce the Web UI Image Change Captioning (WUICC) task, which replaces binary or highlight-based alerts with semantic natural language descriptions, and present WUICC-bench—the first dedicated dataset for this purpose. Evaluations of eleven image-difference captioning methods and two zero-shot large models demonstrate the feasibility of semantic VRT in scenarios involving diverse layouts, text-heavy content, and fine-grained changes. Although current approaches remain challenging, they significantly outperform pixel-wise comparison and exhibit superior selectivity in suppressing irrelevant visual noise, thereby laying the groundwork for future domain-specific model development.

Abstract

Visual regression testing (VRT) is a standard quality assurance step in modern software release pipelines. On every change, it re-renders user interface (UI) screenshots, compares each one against an approved baseline image, and routes any detected difference to a human reviewer who decides whether it is an intended update or an unintended regression. A widely used approach, especially in open-source and continuous-integration pipelines, is pixel-level comparison, which is semantically blind and treats rendering noise and genuine defects identically, producing large volumes of false positives that force developers and testers to spend substantial time and effort manually reviewing flagged differences at every release cycle. Industry tools apply machine learning to VRT, but lack public evaluation. More critically, no dataset or benchmark exists to support natural language descriptions of UI changes, a capability that tells testers what changed in words instead of leaving them to interpret a binary flag or a highlighted region. To address the gap, we propose a new task, Web UI Image Change Captioning (WUICC), which sits at the intersection of VRT and image difference captioning (IDC), and release WUICC-bench, its first dataset and benchmark for the task. We evaluate eleven representative IDC methods, together with two zero-shot general-purpose LLMs. We find that: (1) these methods tend to struggle in the Web UI domain due to its layout diversity, dense text, and fine-grained changes, and (2) yet the trained methods already suppress non-meaningful visual noise far more selectively than the pixel-level comparison VRT relies on, providing a solid foundation for future domain-specific research.

Problem

• Visual Regression Testing
• Image Change Captioning
• Web UI
• False Positives
• Natural Language Description

Innovation

• Web UI Image Change Captioning
• Visual Regression Testing
• Image Difference Captioning
• WUICC-bench
• Natural Language Description