ShadAR: LLM-driven shader generation to transform visual perception in Augmented Reality

2026-02-19

AI Summary

This work proposes an end-to-end large language model (LLM)-driven framework that addresses the limitations of existing augmented reality (AR) systems, which rely on predefined visual effects and struggle to accommodate users’ personalized perceptual needs. By interpreting natural language instructions in real time, the framework automatically generates and deploys AR shader code to dynamically adjust visual rendering on head-mounted displays. This approach represents the first integration of LLMs with AR shader generation, enabling users to freely customize their visual experiences through natural language. The system not only enhances accessibility and inclusivity but also expands the boundaries of creative expression in AR, demonstrating the feasibility and effectiveness of real-time, user-driven visual perception adaptation.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Augmented Reality (AR) can simulate various visual perceptions, such as how individuals with colorblindness see the world. However, these simulations require developers to predefine each visual effect, limiting flexibility. We present ShadAR, an AR application enabling real-time transformation of visual perception through shader generation using large language models (LLMs). ShadAR allows users to express their visual intent via natural language, which is interpreted by an LLM to generate corresponding shader code. This shader is then compiled real-time to modify the AR headset viewport. We present our LLM-driven shader generation pipeline and demonstrate its ability to transform visual perception for inclusiveness and creativity.

Problem

• Augmented Reality
• visual perception
• shader generation
• colorblindness simulation
• real-time customization

Innovation

• LLM-driven shader generation
• Augmented Reality
• visual perception transformation
• real-time shader compilation
• natural language interface