Task Me Anything

2024-06-17

arXiv.org

AI Summary

Existing multimodal LLM (MLM) evaluation benchmarks inadequately assess fine-grained capabilities—such as object recognition, attribute understanding, and spatiotemporal or relational reasoning—in realistic user-facing scenarios. To address this, we propose the first user-query-driven dynamic benchmark generation paradigm. Our approach integrates taxonomy-aware multimodal asset classification, procedural task synthesis, constraint-aware query optimization, and cross-modal automatic alignment annotation, enabling scalable construction of a hierarchical multimodal asset taxonomy and a million-scale controllable QA-pair generation algorithm. The system supports three modalities—images, videos, and 3D objects—and generates 750 million high-quality QA pairs. Empirically, it reveals, for the first time, systematic structural deficiencies of open-source MLMs in spatiotemporal reasoning, precisely characterizing their capability boundaries. This facilitates scenario-driven model selection and targeted optimization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Benchmarks for large multimodal language models (MLMs) now serve to simultaneously assess the general capabilities of models instead of evaluating for a specific capability. As a result, when a developer wants to identify which models to use for their application, they are overwhelmed by the number of benchmarks and remain uncertain about which benchmark's results are most reflective of their specific use case. This paper introduces Task-Me-Anything, a benchmark generation engine which produces a benchmark tailored to a user's needs. Task-Me-Anything maintains an extendable taxonomy of visual assets and can programmatically generate a vast number of task instances. Additionally, it algorithmically addresses user queries regarding MLM performance efficiently within a computational budget. It contains 113K images, 10K videos, 2K 3D object assets, over 365 object categories, 655 attributes, and 335 relationships. It can generate 750M image/video question-answering pairs, which focus on evaluating MLM perceptual capabilities. Task-Me-Anything reveals critical insights: open-source MLMs excel in object and attribute recognition but lack spatial and temporal understanding; each model exhibits unique strengths and weaknesses; larger models generally perform better, though exceptions exist; and GPT4o demonstrates challenges in recognizing rotating/moving objects and distinguishing colors.

Problem

• Multimodal Language Models
• Comprehensive Assessment
• Object Recognition

Innovation

• Task-Me-Anything
• Multimodal Language Model Evaluation
• Visual Understanding Assessment