HALO: Human-Aligned End-to-end Image Retargeting with Layered Transformations

2025-04-03

AI Summary

Image retargeting aims to preserve content integrity and structural consistency during aspect-ratio adaptation, yet existing methods often introduce visual artifacts or distort semantic structure. To address this, we propose an end-to-end trainable framework featuring a novel hierarchical deformation field grounded in human visual sensitivity: rigid constraints are enforced on salient regions, while non-salient regions undergo adaptive, content-aware warping. We further introduce a perception-driven structural similarity loss (P-SSIM) that jointly optimizes content fidelity, geometric alignment, and perceptual consistency. Our approach integrates hierarchical image decomposition, differentiable deformation modeling, and multi-objective optimization. Evaluated on the RetargetMe benchmark, our method achieves state-of-the-art performance, with an average 18.4% improvement in user preference over the best prior baseline.

Technology Category

• Computer Vision: Image and Video Retrieval
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Image retargeting aims to change the aspect-ratio of an image while maintaining its content and structure with less visual artifacts. Existing methods still generate many artifacts or fail to maintain original content or structure. To address this, we introduce HALO, an end-to-end trainable solution for image retargeting. Since humans are more sensitive to distortions in salient areas than non-salient areas of an image, HALO decomposes the input image into salient/non-salient layers and applies different wrapping fields to different layers. To further minimize the structure distortion in the output images, we propose perceptual structure similarity loss which measures the structure similarity between input and output images and aligns with human perception. Both quantitative results and a user study on the RetargetMe dataset show that HALO achieves SOTA. Especially, our method achieves an 18.4% higher user preference compared to the baselines on average.

Problem

• Reduces visual artifacts in image retargeting
• Preserves content and structure in salient areas
• Aligns output with human perception metrics

Innovation

• End-to-end trainable image retargeting solution
• Layered transformation for salient and non-salient areas
• Perceptual structure similarity loss for alignment