Conversational Tactile Data Interfaces: Co-Designing Accessible Data Experiences with Blind Users Using Refreshable Tactile Displays and Conversational AI

2026-07-16

AI Summary

This study addresses the challenge of enabling blind and low-vision users to explore complex data through an accessible, multimodal interface that effectively integrates refreshable tactile displays (RTDs) with large language model (LLM)-driven conversational AI. Through participatory co-design and Wizard-of-Oz prototyping, the authors developed Graphy, a tactile data interaction system that introduces a hierarchical tactile rendering mechanism, a syntactic distinction between user- and agent-initiated haptic feedback, and a structured “select–confirm–query–verify” interaction protocol. By synergistically combining tactile and linguistic modalities, Graphy significantly enhances users’ spatial comprehension and verification of intricate data structures, trends, and relationships. The work culminates in a set of core design principles for tactile-conversational data interfaces that prioritize accessibility, expressiveness, and interactive coherence.

Abstract

Combining refreshable tactile displays (RTDs) with conversational AI offers a promising approach to accessible data visualization for people who are blind or have low vision (BLV). However, it remains an open question how these modalities should be integrated to support accessible data experiences. We address this through a co-design process with three BLV co-designers. Building on our prior Wizard-of-Oz study, we created a conversational tactile data interface (CTDI) that combines an RTD with an LLM-powered conversational agent, refined through four workshops over eight months. In addition to the resulting system, Graphy, we contribute design knowledge and recommendations for CTDIs. Co-designers used touch as the primary sensemaking channel for spatial understanding of the data's shape, trends, and relationships, reserved the agent for what touch could not resolve (e.g., calculation and analysis), and used the chart on the RTD to verify the agent's responses. Key findings include: a layered presentation that scaffolds chart exploration through progressive, interactive layers; a feedback grammar that distinguishes user- and agent-initiated tactile feedback; and a sequential interaction pattern -- select, confirm, ask, verify -- where each step grounds the last.

Problem

• accessible data visualization
• refreshable tactile displays
• conversational AI
• blind users
• tactile data interfaces

Innovation

• Conversational Tactile Data Interface
• Refreshable Tactile Display
• Co-Design
• Accessible Data Visualization
• Multimodal Interaction