VIBES: Exploring Viewer Spatial Interactions as Direct Input for Livestreamed Content

2025-04-12

AI Summary

Live-streaming audiences are currently limited to text-based chat for interaction, lacking real-time influence over content. This paper introduces VIBES, the first system to enable fine-grained spatial interaction—such as mouse clicks and cursor movement—in live-streaming contexts, allowing viewers to manipulate video frames in real time through embodied, low-latency control. Built on web-based coordinate capture, ultra-low-latency command transmission, and a modular, plug-and-play API architecture, VIBES is lightweight and compatible with mainstream streaming and playback infrastructures. Unlike conventional text-only interaction, spatial interaction transcends unidirectional, low-bandwidth paradigms, enabling collaborative co-creation, enhancing主播 controllability, and strengthening cross-user social cohesion. A field study conducted during live broadcasts demonstrates statistically significant improvements in viewer session duration, interaction frequency, and perceived community belongingness.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Computer Vision: Video Understanding & Activity Analysis
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Economics, Online Markets and Human Computation: The sharing economy
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search

Abstract

Livestreaming has rapidly become a popular online pastime, with real-time interaction between streamer and viewer being a key motivating feature. However, viewers have traditionally had limited opportunity to directly influence the streamed content; even when such interactions are possible, it has been reliant on text-based chat. We investigate the potential of spatial interaction on the livestreamed video content as a form of direct, real-time input for livestreamed applications. We developed VIBES, a flexible digital system that registers viewers' mouse interactions on the streamed video, i.e., clicks or movements, and transmits it directly into the streamed application. We used VIBES as a technology probe; first designing possible demonstrative interactions and using these interactions to explore streamers' perception of viewer influence and possible challenges and opportunities. We then deployed applications built using VIBES in two livestreams to explore its effects on audience engagement and investigate their relationships with the stream, the streamer, and fellow audience members. The use of spatial interactions enhances engagement and participation and opens up new avenues for both streamer-viewer and viewer-viewer participation. We contextualize our findings around a broader understanding of motivations and engagement in livestreaming, and we propose design guidelines and extensions for future research.

Problem

• Enabling direct spatial viewer input in livestreams
• Exploring real-time interaction beyond text-based chat
• Enhancing engagement via spatial interactions in streaming

Innovation

• Spatial interaction as direct livestream input
• Mouse movements and clicks influence content
• Enhances engagement via viewer participation