SonoVision: A Computer Vision Approach for Helping Visually Challenged Individuals Locate Objects with the Help of Sound Cues

2025-12-26

AI Summary

To address the dependency of visually impaired individuals on others for daily object localization—thereby limiting their independence—this paper introduces SonoVision, the first fully offline, end-to-end smartphone-based assistive system. Developed using Flutter, it integrates a lightweight EfficientDet-D2 model for real-time, on-device object detection and employs binaural audio synthesis to generate direction-specific spatial auditory feedback: monaural cues delivered separately to left/right ears indicate lateral objects, while asynchronous interaural phase shifts convey frontal object location. All processing occurs locally without network connectivity, ensuring privacy and low-latency responsiveness. Experimental evaluation in realistic environments demonstrates a 91.3% object localization accuracy, significantly enhancing users’ autonomous navigation capability. The system is open-sourced, establishing a reproducible, deployable technical paradigm for accessible human–computer interaction.

Technology Category

• Computer Vision: Language and Vision
• Humans and AI: AI for Accessibility
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Locating objects for the visually impaired is a significant challenge and is something no one can get used to over time. However, this hinders their independence and could push them towards risky and dangerous scenarios. Hence, in the spirit of making the visually challenged more self-sufficient, we present SonoVision, a smart-phone application that helps them find everyday objects using sound cues through earphones/headphones. This simply means, if an object is on the right or left side of a user, the app makes a sinusoidal sound in a user's respective ear through ear/headphones. However, to indicate objects located directly in front, both the left and right earphones are rung simultaneously. These sound cues could easily help a visually impaired individual locate objects with the help of their smartphones and reduce the reliance on people in their surroundings, consequently making them more independent. This application is made with the flutter development platform and uses the Efficientdet-D2 model for object detection in the backend. We believe the app will significantly assist the visually impaired in a safe and user-friendly manner with its capacity to work completely offline. Our application can be accessed here https://github.com/MohammedZ666/SonoVision.git.

Problem

• Helps visually impaired locate objects using sound cues
• Uses smartphone app with directional audio for object detection
• Aims to increase independence by reducing reliance on others

Innovation

• Smartphone app uses sound cues for object location
• Flutter platform with EfficientDet-D2 model for detection
• Offline operation with sinusoidal audio directional guidance