A Multimodal Assistive System for Product Localization and Retrieval for People who are Blind or have Low Vision

2026-01-18

AI Summary

This work addresses the challenge faced by blind or low-vision users in independently locating and retrieving products in physical retail environments. To this end, the authors propose a multimodal wearable assistive system that integrates YOLO-World object detection, a vision-language model (VLM), embedding similarity computation, and color histogram matching within a closed-loop “search–navigate–verify” pipeline. Real-time guidance is delivered through spatialized audio cues. The system represents the first end-to-end application of this synergistic combination of techniques for product localization and verification. It achieves high shelf detection accuracy within 1.5 meters and near-perfect product identification at close range. Moreover, VLM-driven navigation and verification attain accuracies of 94.4% and over 86%, respectively, substantially enhancing users’ autonomy in shopping tasks.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Ad search and search for Web retail
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Shopping is a routine activity for sighted individuals, yet for people who are blind or have low vision (pBLV), locating and retrieving products in physical environments remains a challenge. This paper presents a multimodal wearable assistive system that integrates object detection with vision-language models to support independent product or item retrieval, with the goal of enhancing users'autonomy and sense of agency. The system operates through three phases: product search, which identifies target products using YOLO-World detection combined with embedding similarity and color histogram matching; product navigation, which provides spatialized sonification and VLM-generated verbal descriptions to guide users toward the target; and product correction, which verifies whether the user has reached the correct product and provides corrective feedback when necessary. Technical evaluation demonstrated promising performance across all modules, with product detection achieving near-perfect accuracy at close range and high accuracy when facing shelves within 1.5 m. VLM-based navigation achieved up to 94.4% accuracy, and correction accuracy exceeded 86% under optimal model configurations. These results demonstrate the system's potential to address the last-meter problem in assistive shopping. Future work will focus on user studies with pBLV participants and integration with multi-scale navigation ecosystems.

Problem

• product localization
• assistive technology
• blindness
• low vision
• shopping assistance

Innovation

• multimodal assistive system
• vision-language model (VLM)
• YOLO-World
• spatialized sonification
• last-meter problem