Task-Specific Multimodal Question Answering Agents via Confidence Calibration and Incremental Reasoning for QANTA 2026

2026-07-10

AI Summary

This work addresses the challenge of jointly optimizing response timing and answer accuracy in resource-constrained, pyramid-style multimodal question answering—comprising Tossup and Bonus subtasks—by introducing a lightweight dual-agent architecture. The Tossup agent dynamically determines when to buzz in through confidence calibration and domain-specific numerical reasoning, while the Bonus agent enhances answer precision by integrating prompt-aware processing, structured relational reasoning, and multimodal evidence fusion. Notably, the approach eschews external retrieval and model ensembling, relying solely on hosted models (GPT-4o-mini-class and GPT-4o-class). Evaluated on the QANTA 2026 challenge, it achieves a winning total score of 0.402 (Tossup: 0.238; Bonus Effect: 0.164), demonstrating the efficacy and efficiency of task-tailored reasoning strategies.

Abstract

We present our submission to the QANTA 2026 shared challenge at the ICML 2026 Workshop on Efficient Multimodal Question Answering (EMM-QA). Quanta evaluates multimodal quizbowl systems that answer pyramid-style questions from incrementally revealed text and accompanying images while operating under realistic efficiency constraints. The challenge consists of two distinct tasks: Tossup questions, which require deciding when to answer under uncertainty, and Bonus questions, which emphasize accurate answer selection and human adoption. To address these differing objectives, we develop a task-specific two-agent architecture. Our Tossup agent utilizes a GPT-4o-mini-class model (referred to as GPT-4.1-mini in the competition logs) with confidence-calibrated answering and a domain-specific numeric reasoning policy that reduces overconfident predictions from isolated quantitative clues. Our Bonus agent uses GPT-4o-class model (referred to as GPT-4.1) with leadin-aware reasoning, structured relational reasoning, and multimodal evidence integration to improve exact answer selection. Rather than relying on a retrieval pipeline or model ensembles, our approach emphasizes efficient reasoning policies and confidence calibration within a hosted-only environment. Our system achieved the highest overall leaderboard score of 0.402, including a Tossup score of 0.238 and a Bonus Effect score of 0.164. The results demonstrate that lightweight, task-specific reasoning strategies can provide strong performance on resource-constrained multimodal question answering benchmarks.

Problem

• Multimodal Question Answering
• Tossup Questions
• Bonus Questions
• Confidence Calibration
• Efficiency Constraints

Innovation

• confidence calibration
• incremental reasoning
• task-specific agents
• multimodal question answering
• efficient reasoning