Streaming, Fast and Slow: Cognitive Load-Aware Streaming for Efficient LLM Serving

2025-04-25

AI Summary

To address computational inefficiency arising from the mismatch between LLM streaming generation rate and human reading speed as well as semantic cognitive load, this paper proposes the first real-time cognitive load–aware adaptive streaming rate control method. It employs a lightweight semantic complexity model to dynamically estimate per-token cognitive load and adjusts inter-token emission intervals accordingly. Integrated with an online flow-control policy and a crowdsourced user evaluation framework, the approach optimizes server-side computation while preserving user experience. Its core innovation lies in introducing fine-grained cognitive load estimation into the LLM streaming service loop—breaking away from conventional fixed or heuristic streaming rate paradigms. Experiments demonstrate that the method reduces server-side computational consumption by 16.8% in multi-user scenarios, significantly improving resource utilization and system scalability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Generative conversational interfaces powered by large language models (LLMs) typically stream output token-by-token at a rate determined by computational budget, often neglecting actual human reading speeds and the cognitive load associated with the content. This mismatch frequently leads to inefficient use of computational resources. For example, in cloud-based services, streaming content faster than users can read appears unnecessary, resulting in wasted computational resources and potential delays for other users, particularly during peak usage periods. To address this issue, we propose an adaptive streaming method that dynamically adjusts the pacing of LLM streaming output in real-time based on inferred cognitive load. Our approach estimates the cognitive load associated with streaming content and strategically slows down the stream during complex or information-rich segments, thereby freeing computational resources for other users. Our statistical analysis of computational savings, combined with crowdsourced user studies, provides insights into the trade-offs between service efficiency and user satisfaction, demonstrating that our method can significantly reduce computational consumption up to 16.8%. This context-aware computational resource management strategy presents a practical framework for enhancing system efficiency in cloud-based conversational AI interfaces without compromising user experience.

Problem

• Mismatch between LLM streaming speed and human reading speed
• Inefficient computational resource use in cloud-based LLM services
• Lack of adaptive streaming based on user cognitive load

Innovation

• Adaptive streaming adjusts LLM output pacing
• Estimates cognitive load for dynamic adjustments
• Reduces computational consumption by 16.8%