Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report

2024-06-17

arXiv.org

AI Summary

Multimodal foundation models (MMFMs) suffer from degraded performance on demanding cross-modal integration tasks—such as document understanding—and incur substantially higher fine-tuning and deployment costs than unimodal counterparts. To address this, we propose a structured generation framework for *frozen* MMFMs that requires no parameter updates: it enforces syntax-compliant structured outputs at inference time via logit-level hard-constrained decoding, implicitly inducing multi-step reasoning; combined with lightweight template design, it enables zero-shot adaptation of frozen models (e.g., Qwen-VL, LLaVA). This approach replaces costly fine-tuning with minimal engineering overhead. Evaluated on the CVPR 2024 Second MMFM Challenge, our method outperforms leading fine-tuned baselines. All code, deployment pipelines, and evaluation results are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Multimodal Foundation Models (MMFMs) have demonstrated strong performance in both computer vision and natural language processing tasks. However, their performance diminishes in tasks that require a high degree of integration between these modalities, such as document understanding. Moreover, finetuning these models and deploying them requires significantly more compute and more engineering effort than unimodal models. In this work, we present Multimodal Structured Generation, a framework that forces (frozen) MMFMs to produce outputs in a strictly structured format by applying hard constraints directly to the output logits. This approach not only ensures that the model generates parseable outputs that downstream APIs can easily ingest but also allows us to force the model to reason before answering, which significantly boosts performance without the need for expensive fine-tuning. We demonstrate the effectiveness of our method through competitive results in the CVPR 2nd MMFM Challenge, highlighting that carefully designed lightweight engineering can outperform expensive and complicated modeling approaches. All of our scripts, deployment steps, and evaluation results can be accessed in https://github.com/leloykun/MMFM-Challenge

Problem

• Enhance multimodal model integration for document understanding.
• Reduce compute and engineering effort for model deployment.
• Ensure structured, parseable outputs from frozen multimodal models.

Innovation

• Multimodal Structured Generation framework
• Hard constraints on output logits
• Lightweight engineering outperforms complex models