Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making

2026-07-15

AI Summary

This work addresses the challenge that large language models (LLMs) struggle to dynamically decide during inference whether to continue reasoning, seek help, invoke tools, or abstain, with existing approaches relying on input-side signals and incurring high maintenance costs. The authors propose a lightweight multi-head control mechanism that, for the first time, directly generates real-time control signals from the hidden-state trajectories of frozen LLMs or vision-language models—without requiring fine-tuning—enabling behavioral modulation at deployment. The framework jointly employs a capability head (to determine if a more powerful model should take over) and a parsing head (to select among clarification, tool invocation, abstention, or direct answering), supporting post-hoc adaptation without altering the backbone model. Experiments on AndroidWorld show a 90.7% reduction in expensive LLM calls, up to a 158% improvement in tool-calling scores, and a 65.5% decrease in missed tool invocations, substantially enhancing the quality–cost trade-off.

Abstract

Large language models are increasingly deployed as agents, but reliable agentic behavior requires more than next-token prediction. At inference time, it is preferred that an agent can decide whether to proceed with its current reasoning, defer to a stronger model, request additional information, invoke external tools, or abstain under the given setup. Existing approaches address these decisions through prompt-level routing, external orchestration, or task-specific fine-tuning, which primarily rely on input-side signals, and are often costly and difficult to maintain as model backbones evolve. We ask whether such control decisions can be inferred directly from a model's latent generation process. We introduce Multi-Head Latent Control, a lightweight layer that reads hidden-state trajectories from a frozen LLM or VLM to produce deployment-time control signals. A Capability Head predicts whether the current model can solve the instance or should defer to a stronger collaborator, while a Resolution Head predicts appropriate resolution decision Clarification, Tool Use, Abstention, or Direct Answering. Both heads are trained only on latent traces from the same frozen LLM backbone, enabling post hoc adaptation without modifying the model. Across language and vision-language settings, Multi-Head Latent Control consistently improves the quality-cost tradeoff of multi-model systems, enabling early handoff from partial generations and more accurate intervention decisions. In routed execution (small + large model), it reduces large-model usage by up to 90.7 percent on AndroidWorld and 27-53 percent on average across benchmarks, while retaining most of large-model performance. Additionally, the learned control signals improve tool-use decision quality, yielding up to +158 percent relative score gain and 65.5 percent fewer missed-required tool calls.

Problem

• LLM agent
• decision making
• control signals
• model routing
• tool use

Innovation

• Multi-Head Latent Control
• Frozen LLM
• Latent-State Routing
• Agentic Decision Making
• Model Handoff