VisKnow: Constructing Visual Knowledge Base for Object Understanding

2025-12-08

AI Summary

Traditional object recognition methods yield only class labels, lacking structured semantic understanding. To address this limitation, this paper proposes VisKnow—a novel framework for constructing multimodal visual knowledge graphs tailored to object-level understanding. Methodologically, VisKnow integrates百科-style textual triple extraction, fine-grained image region annotation, and cross-modal alignment, leveraging large language models and vision models to automatically extract and structurally model part-, attribute-, and context-level knowledge. Its core contribution is AnimalKB: a comprehensive knowledge base covering 406 animal categories, comprising 22K textual triples and 420K annotated images. Experimental results demonstrate significant improvements in zero-shot recognition and fine-grained visual question answering; moreover, AnimalKB enables novel downstream tasks such as knowledge graph completion. By unifying visual perception with structured, interpretable, and reasoning-capable representations, VisKnow establishes foundational infrastructure for explainable and inferential vision understanding.

Technology Category

• Computer Vision: Multi-modal Vision
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Understanding objects is fundamental to computer vision. Beyond object recognition that provides only a category label as typical output, in-depth object understanding represents a comprehensive perception of an object category, involving its components, appearance characteristics, inter-category relationships, contextual background knowledge, etc. Developing such capability requires sufficient multi-modal data, including visual annotations such as parts, attributes, and co-occurrences for specific tasks, as well as textual knowledge to support high-level tasks like reasoning and question answering. However, these data are generally task-oriented and not systematically organized enough to achieve the expected understanding of object categories. In response, we propose the Visual Knowledge Base that structures multi-modal object knowledge as graphs, and present a construction framework named VisKnow that extracts multi-modal, object-level knowledge for object understanding. This framework integrates enriched aligned text and image-source knowledge with region annotations at both object and part levels through a combination of expert design and large-scale model application. As a specific case study, we construct AnimalKB, a structured animal knowledge base covering 406 animal categories, which contains 22K textual knowledge triplets extracted from encyclopedic documents, 420K images, and corresponding region annotations. A series of experiments showcase how AnimalKB enhances object-level visual tasks such as zero-shot recognition and fine-grained VQA, and serves as challenging benchmarks for knowledge graph completion and part segmentation. Our findings highlight the potential of automatically constructing visual knowledge bases to advance visual understanding and its practical applications. The project page is available at https://vipl-vsu.github.io/VisKnow.

Problem

• Constructs visual knowledge base for comprehensive object understanding
• Integrates multi-modal data like images, text, and region annotations
• Enhances tasks such as zero-shot recognition and fine-grained VQA

Innovation

• Structures multi-modal object knowledge as graphs
• Extracts text and image knowledge with region annotations
• Automatically constructs visual knowledge bases for tasks