Patent Figure Classification using Large Vision-language Models

2025-01-22

AI Summary

This work addresses the challenging problem of fine-grained classification of patent drawings—spanning type, projection, patent category, and depicted object—a specialized image understanding task requiring domain-specific visual reasoning. We propose a zero- and few-shot learning framework built upon Large Vision-Language Models (LVLMs), introducing a novel tournament-style multiple-choice question-answering classification strategy to enable structured visual reasoning. To support rigorous evaluation, we construct PatFigVQA/PatFigCLS, the first benchmark dataset specifically designed for patent figures, featuring multi-dimensional annotations and visual question answering (VQA) tasks. Extensive experiments demonstrate that our method significantly outperforms conventional CNN-based baselines under few-shot settings, validating both the efficacy of LVLMs for domain adaptation in technical imagery and the superiority of our proposed reasoning strategy.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Patent figure classification facilitates faceted search in patent retrieval systems, enabling efficient prior art search. Existing approaches have explored patent figure classification for only a single aspect and for aspects with a limited number of concepts. In recent years, large vision-language models (LVLMs) have shown tremendous performance across numerous computer vision downstream tasks, however, they remain unexplored for patent figure classification. Our work explores the efficacy of LVLMs in patent figure visual question answering (VQA) and classification, focusing on zero-shot and few-shot learning scenarios. For this purpose, we introduce new datasets, PatFigVQA and PatFigCLS, for fine-tuning and evaluation regarding multiple aspects of patent figures~(i.e., type, projection, patent class, and objects). For a computational-effective handling of a large number of classes using LVLM, we propose a novel tournament-style classification strategy that leverages a series of multiple-choice questions. Experimental results and comparisons of multiple classification approaches based on LVLMs and Convolutional Neural Networks (CNNs) in few-shot settings show the feasibility of the proposed approaches.

Problem

• Image Text Models
• Patent Image Classification
• Information Retrieval Efficiency

Innovation

• LVLM
• Patent Graphic Classification
• Choice-Format Multiclass Identification