Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance

2026-08-28

AI Summary

为解决视障辅助技术中信息优先级问题，本文提出一种基于显著性的视觉-语言模型Salience-LLaVA，并构建了相关数据集以优化场景描述。

Abstract

Vision-language models (VLMs) are rapidly progressing and offer promising capabilities for assistive technologies supporting persons with blindness or low vision. However, existing VLMs are primarily designed for general-purpose captioning and do not explicitly model human perceptual priorities, thereby limiting their ability to emphasize the most relevant information in a scene. To address this gap, we propose a salience-driven captioning framework that prioritizes scene elements according to their importance for human-centered assistance. We curate three salience-aware datasets, namely, Salience COCO, Salience Flickr, and Salience VizWiz, with object-level salience annotations designed to reflect the visual information most relevant to low vision users across different environments. Building on these datasets, we introduce Salience-LLaVA, a salience-aware VLM that incorporates salience cues to generate captions in which important elements are mentioned in the order of importance. Our work makes four main contributions. We build salience-aware datasets verified by low vision participants, propose Salience-LLaVA to describe objects in the order of importance, introduce SCMI to evaluate ordering accuracy, and deploy the system on assistive glasses to demonstrate real-world practicality. Code and datasets are available at: https://github.com/topo-focus/Topofocus

Problem

• Vision-Language Models
• Low Vision Assistance
• Salience
• Perceptual Priorities
• Captioning

Innovation

• salience-driven
• vision-language model
• low vision assistance
• Salience-LLaVA
• SCMI