FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition

2026-03-27

AI Summary

This work addresses the inefficiency and insufficient robustness of static fusion approaches in multimodal human recognition under unconstrained scenarios by proposing an agent-based framework built upon multimodal large language models (MLLMs). The framework employs reinforcement-based fine-tuning to enable sample-adaptive dynamic model selection and introduces an Anchor-based Confidence Top-k (ACT) score fusion strategy to effectively resolve misalignment among multimodal confidence scores. As the first study to integrate agent mechanisms into multimodal biometric recognition, the proposed method achieves dynamic, interpretable, and efficient model composition. Experimental results demonstrate that the approach significantly outperforms state-of-the-art methods across multiple full-body biometric benchmarks while reducing the number of model invocations, thereby enhancing both recognition efficiency and robustness.

Abstract

Model fusion is a key strategy for robust recognition in unconstrained scenarios, as different models provide complementary strengths. This is especially important for whole-body human recognition, where biometric cues such as face, gait, and body shape vary across samples and are typically integrated via score-fusion. However, existing score-fusion strategies are usually static, invoking all models for every test sample regardless of sample quality or modality reliability. To overcome these limitations, we propose \textbf{FusionAgent}, a novel agentic framework that leverages a Multimodal Large Language Model (MLLM) to perform dynamic, sample-specific model selection. Each expert model is treated as a tool, and through Reinforcement Fine-Tuning (RFT) with a metric-based reward, the agent learns to adaptively determine the optimal model combination for each test input. To address the model score misalignment and embedding heterogeneity, we introduce Anchor-based Confidence Top-k (ACT) score-fusion, which anchors on the most confident model and integrates complementary predictions in a confidence-aware manner. Extensive experiments on multiple whole-body biometric benchmarks demonstrate that FusionAgent significantly outperforms SoTA methods while achieving higher efficiency through fewer model invocations, underscoring the critical role of dynamic, explainable, and robust model fusion in real-world recognition systems. Project page: \href{https://fusionagent.github.io/}{FusionAgent}.

Problem

• model fusion
• human recognition
• dynamic model selection
• multimodal biometrics
• score-fusion

Innovation

• Dynamic Model Selection
• Multimodal Agent
• Reinforcement Fine-Tuning
• Score Fusion
• Whole-body Human Recognition