When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs

2025-09-20

AI Summary

Small vision-language models (S-VLMs) significantly underperform large vision-language models (L-VLMs) on visual question answering (VQA), and existing knowledge transfer methods rely heavily on labeled data. To address this, we propose an unsupervised knowledge alignment framework that requires no human annotations. Our method systematically analyzes discrepancies between L-VLMs and S-VLMs in reasoning paths, attention distributions, and output confidence scores, and constructs a discrepancy-aware alignment loss for targeted knowledge transfer. Crucially, we depart from conventional label-dependent knowledge distillation by introducing differentiable inter-model consistency constraints, thereby enhancing the S-VLM’s capacity to model fine-grained vision–language alignment. Evaluated on four major benchmarks—TextVQA, ST-VQA, ChartQA, and OKVQA—our approach boosts S-VLM performance by an average of 8.2%, substantially narrowing the gap with L-VLMs while retaining over 90% inference efficiency.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Vision-Language Models (L-VLMs) have demonstrated remarkable performance in various vision and language tasks, including visual question answering (VQA). However, their high computational cost makes them impractical for resource-constrained settings and inference-heavy applications. In contrast, Small Vision-Language Models (S-VLMs) offer efficiency but suffer from a significant performance gap compared to their larger counterparts. In this work, we introduce the Model Parity Aligner (MPA), a novel framework designed to systematically improve S-VLMs by leveraging unlabeled images and effective knowledge transfer from L-VLMs. Instead of traditional knowledge distillation methods that rely on labeled training data, MPA employs a strategic parity-based approach that precisely identifies the knowledge disparities between S-VLMs and L-VLMs, and optimizes training by targeting only these disparities. We conduct extensive experiments on four diverse VQA benchmarks, namely TextVQA, ST-VQA, ChartQA, and OKVQA, each of which requires specialized reasoning capabilities such as text recognition, chart interpretation, and commonsense and factual understanding. Our results demonstrate that MPA consistently enhances the performance of S-VLMs on all benchmarks, reducing the performance gap while maintaining computational efficiency. We make our code publicly available.

Problem

• Bridging performance gap between large and small vision-language models
• Enhancing small VLMs' efficiency without labeled training data
• Transferring knowledge from large to small models for VQA tasks

Innovation

• Label-free knowledge transfer from large to small VLMs
• Parity-based approach targeting specific performance gaps
• Systematic improvement using unlabeled images and strategic optimization