RETHINED: A New Benchmark and Baseline for Real-Time High-Resolution Image Inpainting On Edge Devices

2025-03-18

AI Summary

Existing image inpainting methods incur prohibitive computational overhead on high-resolution images and are impractical for edge-device deployment. To address this, we propose the first real-time ultra-high-definition (UHD) image inpainting framework tailored for edge computing. Our approach introduces a lightweight CNN backbone and a resolution-agnostic local patch replacement mechanism—eliminating global upsampling and redundant computations. We further construct DF8K-Inpainting, the first UHD inpainting dataset supporting arbitrary masks. Combined with cross-platform mobile optimization techniques, our system achieves ≤30 ms per-frame inference on mainstream mobile devices—100× faster than state-of-the-art methods—while preserving comparable visual quality. This work significantly advances the practical deployment of high-resolution image inpainting in resource-constrained edge scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Non-convex Optimization

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Existing image inpainting methods have shown impressive completion results for low-resolution images. However, most of these algorithms fail at high resolutions and require powerful hardware, limiting their deployment on edge devices. Motivated by this, we propose the first baseline for REal-Time High-resolution image INpainting on Edge Devices (RETHINED) that is able to inpaint at ultra-high-resolution and can run in real-time ($leq$ 30ms) in a wide variety of mobile devices. A simple, yet effective novel method formed by a lightweight Convolutional Neural Network (CNN) to recover structure, followed by a resolution-agnostic patch replacement mechanism to provide detailed texture. Specially our pipeline leverages the structural capacity of CNN and the high-level detail of patch-based methods, which is a key component for high-resolution image inpainting. To demonstrate the real application of our method, we conduct an extensive analysis on various mobile-friendly devices and demonstrate similar inpainting performance while being $mathrm{100 imes faster}$ than existing state-of-the-art methods. Furthemore, we realease DF8K-Inpainting, the first free-form mask UHD inpainting dataset.

Problem

• High-resolution image inpainting on edge devices
• Real-time performance with low latency
• Lightweight CNN and patch replacement for texture

Innovation

• Lightweight CNN for structure recovery
• Resolution-agnostic patch replacement mechanism
• Real-time high-resolution inpainting on edge devices