Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization

2025-02-28

AI Summary

This study addresses the misalignment between conventional chart similarity metrics and human perceptual judgment in information visualization, proposing a deep feature–based similarity assessment method to enhance visualization retrieval and recommendation systems. Methodologically, it systematically compares five ImageNet-pretrained CNN architectures (e.g., ResNet, VGG) against the traditional MS-SSIM metric, integrating multi-scale structural similarity analysis and crowdsourced perceptual experiments to evaluate performance on scatterplot and visual channel similarity tasks. Key contributions include: (1) the first empirical demonstration that ImageNet-pretrained features significantly outperform fine-tuned MS-SSIM; (2) evidence that high-level semantic features better align with human visual perception than low-level statistical features; and (3) establishment of a transferable, perception-aligned similarity metric foundation for visualization analysis tools.

Technology Category

• Computer Vision: Image and Video Retrieval
• Machine Learning: Learning Preferences or Rankings
• Data Mining & Knowledge Management: Data Visualization & Summarization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data visualization
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Judging the similarity of visualizations is crucial to various applications, such as visualization-based search and visualization recommendation systems. Recent studies show deep-feature-based similarity metrics correlate well with perceptual judgments of image similarity and serve as effective loss functions for tasks like image super-resolution and style transfer. We explore the application of such metrics to judgments of visualization similarity. We extend a similarity metric using five ML architectures and three pre-trained weight sets. We replicate results from previous crowd-sourced studies on scatterplot and visual channel similarity perception. Notably, our metric using pre-trained ImageNet weights outperformed gradient-descent tuned MS-SSIM, a multi-scale similarity metric based on luminance, contrast, and structure. Our work contributes to understanding how deep-feature-based metrics can enhance similarity assessments in visualization, potentially improving visual analysis tools and techniques. Supplementary materials are available at https://osf.io/dj2ms.

Problem

• Explores deep-feature-based similarity metrics for visualization similarity judgments.
• Extends similarity metrics using ML architectures and pre-trained weight sets.
• Improves visualization tools by enhancing similarity assessments with deep-feature metrics.

Innovation

• Deep-feature-based similarity metrics for visualization
• Five ML architectures and pre-trained weights used
• ImageNet weights outperform traditional similarity metrics