MIST: Multimodal Interactive Speech-based Tool-calling Conversational Assistants for Smart Homes

2026-05-07

AI Summary

This work addresses the challenge that existing voice assistants struggle to model spatiotemporal constraints, dynamic states, and mixed-initiative interaction patterns of IoT devices in smart homes. To bridge this gap, we propose a voice-driven, multi-turn code generation task and introduce the first synthetic dataset that integrates spoken-language inputs, physical-world constraints, and mixed-initiative interactions, accompanied by a scalable data generation framework. Building upon large language models, our system unifies multimodal tool invocation, dynamic state tracking, and spoken-language semantic understanding. Experimental results reveal a significant performance disparity between open- and closed-source multimodal large language models on this task. We release the benchmark dataset and an open framework to foster further research in this emerging domain.

Abstract

The rise of Internet of Things (IoT) devices in the physical world necessitates voice-based interfaces capable of handling complex user experiences. While modern Large Language Models (LLMs) already demonstrate strong tool-usage capabilities, modeling real-world IoT devices presents a difficult, understudied challenge which combines modeling spatiotemporal constraints with speech inputs, dynamic state tracking, and mixed-initiative interaction patterns. We introduce MIST (the Multimodal Interactive Speech-based Tool-calling Dataset), a synthetic multi-turn, voice-driven code generation task that operates over IoT devices. We find that there is a significant gap between open- and closed-weight multimodal LLMs on MIST, and that even frontier closed-weight LLMs have substantial headroom. We release MIST and an extensible data generation framework to build related datasets in order to facilitate research on mixed-initiative voice assistants which reason about physical world constraints.

Problem

• IoT
• voice-based interfaces
• spatiotemporal constraints
• mixed-initiative interaction
• dynamic state tracking

Innovation

• multimodal LLMs
• speech-based interaction
• tool calling
• IoT reasoning
• mixed-initiative dialogue