Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games

2026-04-17

AI Summary

This work addresses the challenge that existing open-vocabulary object recognition methods struggle to simultaneously achieve coarse-grained generality and fine-grained discriminative capability. To this end, we propose HyMOR, a hybrid framework that synergistically integrates multimodal large language models (MLLMs) and CLIP for the first time: the MLLM handles open-domain coarse-grained recognition, while CLIP specializes in fine-grained discrimination within domains such as flora and fauna. A Sentence-BERT-guided hybrid reasoning mechanism enables multi-granularity semantic alignment between the two streams. Built upon this architecture, we introduce a unified perception system tailored for educational games and release TBO, a textbook-derived dataset. Our approach achieves a 23.2% improvement in average SBert score, narrows the fine-grained recognition gap to 0.2%, and enhances general recognition performance by 2.5%, substantially strengthening the perceptual foundation for multimodal interactive learning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have enabled open-ended object recognition, yet they struggle with fine-grained tasks. In contrast, CLIP-style models excel at fine-grained recognition but lack broad coverage of general object categories. To bridge this gap, we propose \textbf{HyMOR}, a \textbf{Hy}brid \textbf{M}ulti-granularity open-ended \textbf{O}bject \textbf{R}ecognition framework that integrates an MLLM with a CLIP model. In HyMOR, the MLLM performs open-ended and coarse-grained object recognition, while the CLIP model specializes in fine-grained identification of domain-specific objects such as animals and plants. This hybrid design enables accurate object understanding across multiple semantic granularities, serving as a robust perceptual foundation for downstream multi-modal content generation and interactive gameplay. To support evaluation in content-rich and educational scenarios, we introduce TBO (TextBook Objects), a dataset containing 20,942 images annotated with 8,816 object categories extracted from textbooks. Extensive experiments demonstrate that HyMOR narrows the fine-grained recognition gap with CLIP to 0.2\% while improving general object recognition by 2.5\% over a baseline MLLM, measured by average Sentence-BERT (SBert) similarity. Overall, HyMOR achieves a 23.2\% improvement in average SBert across all evaluated datasets, highlighting its effectiveness in enabling accurate perception for multi-modal game content generation and interactive learning applications.

Problem

• open-ended object recognition
• multi-granularity
• fine-grained recognition
• multimodal perception
• interactive educational games

Innovation

• Hybrid Object Recognition
• Multi-Granularity Perception
• Multimodal Large Language Models
• CLIP Integration
• Educational Game Interaction