Supporting Multimodal Data Interaction on Refreshable Tactile Displays: An Architecture to Combine Touch and Conversational AI

2026-02-16

AI Summary

This work addresses the lack of architectural support for efficient multimodal data interaction by blind or low-vision users via refreshable tactile displays. We propose a novel interaction framework that integrates tactile input with conversational AI, enabling, for the first time on such displays, referential queries grounded in both touch context and natural language. The architecture incorporates key components including external tactile sensing, vision-to-tactile encoding, and synchronized multimodal output, and is accompanied by an open-source reference implementation. By bridging tactile exploration with linguistic interaction, our approach substantially enhances the expressiveness and usability of tactile interfaces, establishing a foundational technical platform for accessible multimodal data visualization.

Technology Category

• Computer Vision: Multi-modal Vision
• Humans and AI: AI for Accessibility
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Combining conversational AI with refreshable tactile displays (RTDs) offers significant potential for creating accessible data visualization for people who are blind or have low vision (BLV). To support researchers and developers building accessible data visualizations with RTDs, we present a multimodal data interaction architecture along with an open-source reference implementation. Our system is the first to combine touch input with a conversational agent on an RTD, enabling deictic queries that fuse touch context with spoken language, such as "what is the trend between these points?" The architecture addresses key technical challenges, including touch sensing on RTDs, visual-to-tactile encoding, integrating touch context with conversational AI, and synchronizing multimodal output. Our contributions are twofold: (1) a technical architecture integrating RTD hardware, external touch sensing, and conversational AI to enable multimodal data interaction; and (2) an open-source reference implementation demonstrating its feasibility. This work provides a technical foundation to support future research in multimodal accessible data visualization.

Problem

• multimodal interaction
• refreshable tactile displays
• accessible data visualization
• conversational AI
• touch input

Innovation

• refreshable tactile displays
• conversational AI
• multimodal interaction
• accessible data visualization
• deictic queries