Measuring User's Mental Models of Speech Translation in Human-AI Collaboration

2026-06-23

AI Summary

This study addresses the challenge that users’ limited understanding of the capability boundaries of speech translation systems hinders effective human–machine collaboration. To investigate how users develop predictive awareness of system errors and refine their mental models, the authors introduce cross-lingual question answering as a novel downstream task, wherein participants answer questions based either on machine-translated output or professionally post-edited translations. Integrating user behavior analysis, speech translation evaluation, and mental model modeling, the research demonstrates that users can improve the accuracy of their mental models through practice—particularly when they possess source-language proficiency, which leads them to rely more on surface-level error cues. Moreover, providing speech transcripts significantly facilitates mental model formation. This work pioneers the use of cross-lingual question answering in mental model research and highlights the critical roles of source-language knowledge and speech transcription.

Abstract

Millions of people use machine translation (MT) tools daily, yet little is known about their perception of what systems can and cannot do. This paper studies users' mental models of speech translation systems through a new framework based on cross-lingual question answering, where users either accept MT output or request professional re-translation to answer questions based on the information presented in a foreign language. By analyzing user behavior and accuracy trends across varying translation qualities, we examine to what extent they can predict where the system is likely to be wrong, and how this mental model evolves. Users develop stronger mental models with practice, especially when they have some knowledge of the source language, primarily by relying on surface-level error cues. Moreover, providing speech transcriptions can help users develop better mental models. Our results show the promise of cross-lingual question answering as a downstream task for studying MT mental models and advancing our understanding of human-AI collaboration.

Problem

• mental models
• speech translation
• human-AI collaboration
• machine translation
• cross-lingual question answering

Innovation

• mental models
• speech translation
• cross-lingual question answering
• human-AI collaboration
• translation quality perception