From GUI Tests to Conversational Interaction: A New Perspective on App-Specific Voice Assistants

2026-07-13

AI Summary

Current voice assistants are predominantly generic systems that struggle to adapt to application-specific behaviors, compelling developers to reimplement interaction logic repeatedly at significant cost. This work proposes a novel approach that leverages GUI test code as a bridge: by employing large language models (LLMs), it automatically parses and refactors Android GUI test scripts to generate tailored voice assistant artifacts—including voice intents, capability descriptions, and executable action plans—without relying on external specifications. The authors implement this method in AppVA, a prototype system evaluated on five open-source Android applications. Experimental results demonstrate that the approach effectively repurposes existing test code to synthesize context-aware, application-specific voice assistants automatically.

Abstract

Voice assistants are widely deployed on mobile platforms, yet most are designed as system-level services that remain poorly aligned with application-specific behavior. As a result, enabling voice interaction at the app level requires developers to manually reimplement application logic, leading to high development and maintenance costs. We propose an LLM-driven approach to automating the development of app-specific voice assistants by repurposing GUI test code, which encodes behavior-preserving, executable specifications of application functionality. In this paper, we present a perspective in which large language models reinterpret GUI tests as bridges between application behavior and conversational interaction. By transforming test methods into app-specific VA artifacts, such as voice intents, capability descriptions, and executable action plans, our approach grounds voice assistants directly in existing application logic rather than external specifications. We illustrate this vision through AppVA, a research prototype on Android. Our preliminary results across five open-source applications suggest that GUI test code can be reused beyond testing, enabling the synthesis of app-specific voice assistants and highlighting a broader research direction at the intersection of software testing, interaction design, and LLM-enabled automation.

Problem

• voice assistants
• application-specific behavior
• GUI tests
• development cost
• mobile platforms

Innovation

• LLM-driven automation
• GUI test reuse
• app-specific voice assistants
• conversational interaction
• behavior-preserving specification