Semantic Similarity is a Spurious Measure of Comic Understanding: Lessons Learned from Hallucinations in a Benchmarking Experiment

2026-03-02

AI Summary

This study addresses the pervasive hallucination issues in current generative vision-language models (VLMs) when performing page-level semantic understanding of comics, which critically hinders effective access for visually impaired users. The work presents the first systematic taxonomy of hallucinations specific to comic understanding and introduces a novel evaluation benchmark tailored for accessibility applications. Through human-in-the-loop analysis, it reveals significant deficiencies in existing models’ semantic coherence and contextual reasoning capabilities, while also highlighting the inadequacy of relying solely on semantic similarity metrics for evaluation. Building on these insights, the authors propose targeted data refinement and hallucination mitigation strategies, laying a foundational framework for developing reliable and interpretable VLMs that support accessible comic comprehension.

Technology Category

• Computer Vision: Large Vision Models
• Humans and AI: AI for Accessibility
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

A system that enables blind or visually impaired users to access comics/manga would introduce a new medium of storytelling to this community. However, no such system currently exists. Generative vision-language models (VLMs) have shown promise in describing images and understanding comics, but most research on comic understanding is limited to panel-level analysis. To fully support blind and visually impaired users, greater attention must be paid to page-level understanding and interpretation. In this work, we present a preliminary benchmark of VLM performance on comic interpretation tasks. We identify and categorize hallucinations that emerge during this process, organizing them into generalized object-hallucination taxonomies. We conclude with guidance on future research, emphasizing hallucination mitigation and improved data curation for comic interpretation.

Problem

• comic understanding
• visual-language models
• hallucination
• accessibility
• page-level interpretation

Innovation

• hallucination taxonomy
• comic understanding
• vision-language models
• benchmarking
• accessibility