HAIFAI: Human-AI Interaction for Mental Face Reconstruction

2024-12-09

AI Summary

This study addresses the challenge of reconstructing invisible facial visual representations encoded in the human brain. We propose a two-stage human–computer collaborative mental face reconstruction framework. In Stage I, lightweight, iterative crowdsourced image ranking feedback drives preference learning to build user-specific computational models—eliminating the need for large-scale manual annotations. In Stage II, diffusion-based generation is integrated with GAN-based facial editing to enable interactive, slider-based fine-tuning. Our approach introduces the first closed-loop human–computer interaction paradigm for mental face reconstruction, substantially reducing data dependency. A user study demonstrates a 60.6% recognition rate—surpassing current state-of-the-art methods—and shows significant improvements in reconstruction fidelity, usability, subjective workload, and interaction efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Crowd Sourcing and Human Computation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We present HAIFAI - a novel two-stage system where humans and AI interact to tackle the challenging task of reconstructing a visual representation of a face that exists only in a person's mind. In the first stage, users iteratively rank images our reconstruction system presents based on their resemblance to a mental image. These rankings, in turn, allow the system to extract relevant image features, fuse them into a unified feature vector, and use a generative model to produce an initial reconstruction of the mental image. The second stage leverages an existing face editing method, allowing users to manually refine and further improve this reconstruction using an easy-to-use slider interface for face shape manipulation. To avoid the need for tedious human data collection for training the reconstruction system, we introduce a computational user model of human ranking behaviour. For this, we collected a small face ranking dataset through an online crowd-sourcing study containing data from 275 participants. We evaluate HAIFAI and an ablated version in a 12-participant user study and demonstrate that our approach outperforms the previous state of the art regarding reconstruction quality, usability, perceived workload, and reconstruction speed. We further validate the reconstructions in a subsequent face ranking study with 18 participants and show that HAIFAI achieves a new state-of-the-art identification rate of 60.6%. These findings represent a significant advancement towards developing new interactive intelligent systems capable of reliably and effortlessly reconstructing a user's mental image.

Problem

• Develops human-AI interaction for mental face reconstruction.
• Introduces computational user model for ranking behavior.
• Enhances reconstruction quality, usability, and speed.

Innovation

• Two-stage human-AI interaction system
• Generative model for initial face reconstruction
• Computational user model for ranking behavior