Augmenting Image Annotation: A Human-LMM Collaborative Framework for Efficient Object Selection and Label Generation

2025-03-14

AI Summary

To address the low efficiency and high cognitive fatigue associated with manual image annotation, this paper proposes a human-AI collaborative annotation framework: humans provide only bounding boxes around target regions, while large multimodal models (e.g., GPT-4V) generate semantic labels automatically via context-aware reasoning conditioned on the bounding boxes. This framework introduces the first “human-boxing + AI-labeling” decoupled division-of-labor paradigm, significantly lightening the annotation workflow. It supports diverse tasks—including object detection, scene description, and fine-grained classification—and enables bidirectional semantic alignment between human and AI through interactive prompt engineering. Experiments demonstrate substantial improvements in annotation throughput, with marked reductions in human cognitive load and annotation time. The approach offers a scalable, low-fatigue solution for large-scale computer vision dataset curation.

Technology Category

• Computer Vision: Multi-modal Vision
• Humans and AI: Crowd Sourcing and Human Computation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Traditional image annotation tasks rely heavily on human effort for object selection and label assignment, making the process time-consuming and prone to decreased efficiency as annotators experience fatigue after extensive work. This paper introduces a novel framework that leverages the visual understanding capabilities of large multimodal models (LMMs), particularly GPT, to assist annotation workflows. In our proposed approach, human annotators focus on selecting objects via bounding boxes, while the LMM autonomously generates relevant labels. This human-AI collaborative framework enhances annotation efficiency by reducing the cognitive and time burden on human annotators. By analyzing the system's performance across various types of annotation tasks, we demonstrate its ability to generalize to tasks such as object recognition, scene description, and fine-grained categorization. Our proposed framework highlights the potential of this approach to redefine annotation workflows, offering a scalable and efficient solution for large-scale data labeling in computer vision. Finally, we discuss how integrating LMMs into the annotation pipeline can advance bidirectional human-AI alignment, as well as the challenges of alleviating the"endless annotation"burden in the face of information overload by shifting some of the work to AI.

Problem

• Reduces human effort in image annotation tasks
• Leverages LMMs for efficient label generation
• Enhances scalability in large-scale data labeling

Innovation

• Human-LMM collaboration for image annotation
• LMM autonomously generates object labels
• Reduces human cognitive and time burden