A11yShape: AI-Assisted 3-D Modeling for Blind and Low-Vision Programmers

2025-08-05

AI Summary

Blind and low-vision (BLV) programmers face significant barriers in understanding, modifying, and authoring 3D models due to the inherent geometric complexity of 3D structures and the lack of non-visual interaction support in existing CAD tools. Method: We present VisuoTactileCAD—a multimodal programming-based modeling system that integrates large language models (LLMs) with OpenSCAD. Its core innovation is a cross-representation highlighting mechanism enabling synchronized semantic alignment among source code, hierarchical semantic structure, AI-generated natural language descriptions, and 3D rendered views. The system further supports accessible natural language interaction, version control, and component-based navigation. Contribution/Results: Four user studies demonstrate that BLV participants can independently complete end-to-end 3D modeling tasks, producing outputs consistent with specifications—marking the first demonstration of fully sighted-assistant-free, non-visual 3D programming.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: AI for Accessibility

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Building 3-D models is challenging for blind and low-vision (BLV) users due to the inherent complexity of 3-D models and the lack of support for non-visual interaction in existing tools. To address this issue, we introduce A11yShape, a novel system designed to help BLV users who possess basic programming skills understand, modify, and iterate on 3-D models. A11yShape leverages LLMs and integrates with OpenSCAD, a popular open-source editor that generates 3-D models from code. Key functionalities of A11yShape include accessible descriptions of 3-D models, version control to track changes in models and code, and a hierarchical representation of model components. Most importantly, A11yShape employs a cross-representation highlighting mechanism to synchronize semantic selections across all model representations -- code, semantic hierarchy, AI description, and 3-D rendering. We conducted a multi-session user study with four BLV programmers, where, after an initial tutorial session, participants independently completed 12 distinct models across two testing sessions, achieving results that aligned with their own satisfaction. The result demonstrates that participants were able to comprehend provided 3-D models, as well as independently create and modify 3-D models -- tasks that were previously impossible without assistance from sighted individuals.

Problem

• Enables BLV users to create 3D models via programming
• Provides non-visual interaction for 3D modeling tasks
• Synchronizes code, hierarchy, and descriptions for accessibility

Innovation

• AI-assisted 3-D modeling for BLV users
• LLM integration with OpenSCAD editor
• Cross-representation highlighting synchronization