MHE-Former: Multi-Hypothesis Transformers via Entropy Maximization for 3D Mesh Recovery

2026-09-09

AI Summary

本文针对单目3D手和身体网格恢复中的遮挡和模糊问题，提出基于熵最大化和Transformer的多假设方法MHE-Former，并通过上下文感知的选择过程提高预测准确性和多样性。

Abstract

Monocular 3D hand and body mesh recovery often suffers from severe occlusion and ambiguity. Traditional deterministic methods typically regress a single optimal solution, leading to overconfident predictions. In this paper, we introduce an exploration--exploitation paradigm for ambiguous mesh recovery with multi-hypothesis learning and selection. Specifically, during exploration, based on our probabilistic formulation and entropy maximization, we propose a novel multi-hypothesis method referred to as MHE-Former. It is a Transformer-based multi-hypothesis framework, ensuring high training efficiency and label friendliness while generating plausible and diverse hypotheses. During exploitation, we propose Hypothesis Selection, a context-aware process for multiple predictions. Especially leveraging VLM's powerful visual understanding and reasoning capabilities, it allows users to choose the most plausible and desired estimate with additional evidence and natural language intent. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in accuracy and diversity across multiple datasets. The user preference study further shows the practicality of our hypothesis selection process.

Problem

• Monocular 3D Mesh Recovery
• Occlusion
• Ambiguity
• Overconfident Predictions

Innovation

• Multi-Hypothesis Transformers
• Entropy Maximization
• Hypothesis Selection
• Visual Language Models