Beyond Fidelity: Semantic Similarity Assessment in Low-Level Image Processing

2026-04-28

AI Summary

Existing image quality assessment methods primarily emphasize visual fidelity and struggle to capture the preservation of semantic content in low-level image processing. This work formally introduces the task of “semantic similarity” evaluation and proposes a structured semantic representation framework that decouples foreground and background entities. By integrating open-world category and relational modeling, the framework constructs image triplets representing semantic structures and introduces a Triplet Semantic Similarity (T3S) score for quantitative assessment. Experiments on COCO and SPA-Data demonstrate that T3S significantly outperforms existing fidelity-based metrics and semantic-level baselines, offering a more accurate characterization of progressive semantic changes under various image degradations.

Abstract

Low-level image processing has long been evaluated mainly from the perspective of visual fidelity. However, with the rise of deep learning and generative models, processed images may preserve perceptual quality while altering semantic content, making conventional Image Quality Assessment (IQA) insufficient for semantic-level assessment. In this paper, we formalize \textit{Semantic Similarity} as a new evaluation task for low-level image processing, aimed at measuring whether semantic content is preserved after processing. We further present a structured formulation of image semantics based on semantic entities and their relations, and discuss the desired properties and constraints of a valid semantic similarity index. Based on this formulation, we propose Triplet-based Semantic Similarity Score (T3S), which models image semantics through foreground entities, background entities, and relations. T3S combines semantic entity extraction, foreground-background disentanglement, and open-world class/relation modeling. Experiments on COCO and SPA-Data show that T3S consistently outperforms existing fidelity-oriented metrics and representative semantic-level baselines, while better reflecting progressive semantic changes under diverse degradations. These results highlight the importance of semantic assessment in modern low-level vision.

Problem

• Semantic Similarity
• Low-Level Image Processing
• Image Quality Assessment
• Semantic Content Preservation

Innovation

• Semantic Similarity
• Low-Level Image Processing
• Triplet-based Semantic Similarity Score
• Semantic Entity
• Image Quality Assessment