How Much Does It Cost to Answer My Question? Benchmarking Cloud VLM-based VQA Systems

2026-08-07

AI Summary

Existing mobile visual question answering (VQA) systems commonly rely on cloud-based vision-language models (VLMs), yet the impact of client-side image preprocessing on cost, latency, and accuracy remains underexplored. This work introduces VQABench, a benchmark that treats preprocessing strategies as controlled variables and systematically evaluates twelve such methods across three datasets and four commercial cloud VLMs through 95,168 API calls. The study comprehensively assesses their effects on answer quality, transmission payload, token consumption, and latency. Findings reveal that preprocessing efficacy is highly dependent on VLM architecture, API design, and pricing policies, and is not universally beneficial; suboptimal choices can simultaneously increase cost and latency while degrading accuracy. These insights offer critical guidance for real-world deployment of mobile VQA systems.

Abstract

Vision-language models (VLMs) are becoming a practical backend for mobile visual question answering (VQA) systems, enabling smartphones and smart glasses to answer users' questions about the physical world. Since modern VLMs remain difficult to run on mobile and edge devices, VQA systems increasingly offload inference to cloud-based VLMs. This gives mobile devices access to stronger computation, but it also makes visual input preparation a key system variable: how the image is prepared before offloading affects not only answer quality but also payload size, token cost, and system latency. Proprietary APIs expose little control over model internals or serving behavior, leaving client-side preprocessing as the main practical optimization space for downstream developers. Many such techniques have been proposed for visual offloading, yet their cost-quality impact on commercial cloud VLMs has never been studied. To fill this gap, we present VQABench, the first systematic benchmark that treats client-side input preprocessing as a controlled variable for cloud-VLM-based VQA. We evaluate 12 preprocessing techniques across three VQA datasets and four commercial VLMs from three providers, totaling 95,168 API calls. Our results show that preprocessing is not universally beneficial: its effectiveness depends on the target model, API paradigm, provider token-accounting rule, and task formulation. A poorly selected preprocessing strategy can increase deployment cost or latency while degrading answer accuracy. Overall, our benchmark clarifies when preprocessing helps, when it fails, and why, providing insights to guide future research and real-world deployment of VQA systems.

Problem

• visual question answering
• vision-language models
• cloud inference
• input preprocessing
• cost-quality trade-off

Innovation

• cloud VLM
• visual question answering
• input preprocessing
• cost-quality tradeoff
• VQABench