AI-Based Speaking Assistant: Supporting Non-Native Speakers' Speaking in Real-Time Multilingual Communication

2025-05-03

AI Summary

Non-native speakers frequently experience reduced communication efficacy in real-time multilingual collaboration due to linguistic imprecision. To address this, we propose the AI-based Spoken-language Assistant (AISA) for non-native speakers. Methodologically, we first empirically identify—via a mixed-methods approach—four prototypical interaction patterns occurring in authentic collaborative settings. We then introduce a novel “logical coherence enhancement–cognitive load balancing” assistance paradigm, integrating context-aware prompt engineering, lightweight speech-to-text transcription, and multi-turn dialogue state modeling within a low-latency communication infrastructure. Experimental results demonstrate that AISA significantly improves users’ spoken logical coherence and content depth, yet does not directly enhance underlying language proficiency. Crucially, we uncover that input and review behaviors induced by the system elevate cognitive load and anxiety—revealing critical human-factor trade-offs. These findings provide empirically grounded design principles for AI-mediated collaborative tools targeting linguistically diverse users.

Technology Category

• Humans and AI: User Experience and Usability
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Non-native speakers (NNSs) often face speaking challenges in real-time multilingual communication, such as struggling to articulate their thoughts. To address this issue, we developed an AI-based speaking assistant (AISA) that provides speaking references for NNSs based on their input queries, task background, and conversation history. To explore NNSs' interaction with AISA and its impact on NNSs' speaking during real-time multilingual communication, we conducted a mixed-method study involving a within-subject experiment and follow-up interviews. In the experiment, two native speakers (NSs) and one NNS formed a team (31 teams in total) and completed two collaborative tasks--one with access to the AISA and one without. Overall, our study revealed four types of AISA input patterns among NNSs, each reflecting different levels of effort and language preferences. Although AISA did not improve NNSs' speaking competence, follow-up interviews revealed that it helped improve the logical flow and depth of their speech. Moreover, the additional multitasking introduced by AISA, such as entering and reviewing system output, potentially elevated NNSs' workload and anxiety. Based on these observations, we discuss the pros and cons of implementing tools to assist NNS in real-time multilingual communication and offer design recommendations.

Problem

• AI assistant aids non-native speakers in real-time multilingual communication
• Explores impact of AI support on speaking flow and depth
• Evaluates workload and anxiety from AI tool multitasking

Innovation

• AI-based real-time speaking assistant for NNS
• Multilingual support with contextual query analysis
• Mixed-method study evaluating impact and workload