Solving Jigsaw Puzzles in the Wild: Human-Guided Reconstruction of Cultural Heritage Fragments

2025-08-31

International Workshop on Machine Learning for Signal Processing

AI Summary

This work addresses the challenge of reconstructing real-world archaeological fragments, which is hindered by weathering, missing parts, irregular shapes, and massive combinatorial ambiguity—particularly when dealing with thousands of pieces, where traditional methods fail. To tackle this, we propose a human-in-the-loop jigsaw framework that integrates an automated relaxed labeling solver with interactive human guidance. Our approach introduces two key innovations: Iterative Anchoring to progressively lock in correct matches and Continuous Interactive Refinement to iteratively correct errors, jointly optimizing both geometric and semantic consistency. Evaluated on the RePAIR dataset, the method substantially outperforms both fully automatic and purely manual baselines, achieving significant gains in both accuracy and efficiency.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Computer Vision: Segmentation
• Constraint Satisfaction and Optimization: Solvers and Tools

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Reassembling real-world archaeological artifacts from fragments is challenging due to erosion, missing regions, irregular shapes, and large-scale ambiguity. Traditional jigsaw solvers, typically designed for clean, synthetic data, struggle especially with thousands of fragments, as in the RePAIR benchmark. We propose a human-in-the-loop (HIL) puzzle-solving framework tailored for real-world cultural heritage reconstruction. Our method combines an automatic relaxation-labeling solver with interactive human guidance, enabling users to iteratively lock verified placements, correct errors, and guide assembly toward semantic and geometric coherence. We introduce two complementary strategies, ie., Iterative Anchoring and Continuous Interactive Refinement, that support scalable reconstruction under varying ambiguity and size. Experiments on RePAIR groups show our hybrid approach significantly outperforms both fully automatic and manual methods in accuracy and efficiency, offering a practical solution for human-in-the-loop artifact reassembly.

Problem

• jigsaw puzzles
• cultural heritage
• fragment reassembly
• archaeological artifacts
• large-scale ambiguity

Innovation

• human-in-the-loop
• cultural heritage reconstruction
• jigsaw puzzle solving
• interactive refinement
• relaxation labeling