StateScribe: Towards Accessible Change Awareness Across Real-World Revisits
career value
206K/year
🤖 AI Summary
This work addresses the challenge faced by blind and low-vision users in detecting dynamic environmental changes across repeated visits, which can lead to safety risks and increased cognitive load. To this end, we propose StateScribe, the first system enabling cross-visit environmental change awareness for this population. StateScribe employs a dual-layer memory architecture that integrates scene-level episodic memory with object-centric temporal memory, coupled with real-time multimodal perception and a lightweight storage mechanism to accurately answer “what changed, where, and when.” Experimental results demonstrate an F1 score of 83.1% over 11 revisits, with average latency under 1.54 seconds and memory usage below 54 MB across 110 revisits. User studies confirm that StateScribe significantly enhances users’ ability to detect environmental changes in real-world settings.
Application Category
📝 Abstract
Real-world environments evolve continuously, yet blind and low-vision (BLV) individuals often have limited access to understanding how they change over time. Unexpected or relocated objects, layout modifications, and content updates (e.g., price changes) can introduce safety risks and cognitive burden. While existing visual assistive technologies can describe immediate surroundings, they operate as one-off interactions and lack mechanisms to surface meaningful changes across revisits. Informed by a survey of 33 BLV individuals, we develop StateScribe, a system that supports accessible awareness of real-world changes across revisits. StateScribe employs a dual-layer memory architecture that integrates episodic scene memory and object-centric temporal memory to enable scalable and structured change tracking. It provides both live descriptions of the current scene, and descriptions of what has changed, when and where it occurred across revisits, such as "The shop on your right has a "CLOSED" sign; it was open at this time last week.'' Our evaluation shows that StateScribe maintains high accuracy (F1-score=83.1%) across 11 revisits, while remaining low-latency (mean<1.54s) and memory-efficient (<54MB) across 110 revisits. A user study with nine BLV participants demonstrates that StateScribe improves change awareness across revisits in three real-world locations. Finally, we discuss implications for long-term AI-assisted companions that support broader change observation using multimodal sensing, extend beyond changes to other memory capabilities, and adapt to individual users, intents, and contexts.
Problem
Research questions and friction points this paper is trying to address.
change awareness
blind and low-vision
real-world revisits
accessible technology
environmental change
Innovation
Methods, ideas, or system contributions that make the work stand out.
dual-layer memory architecture
change awareness
accessible AI
temporal scene understanding
assistive technology for BLV
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
