Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion

2025-05-23

AI Summary

Existing Vision Instruction Tuning (VisIT) datasets rely heavily on closed-source large model APIs and manual construction, suffering from missing documentation, irreproducibility, high cost, and poor scalability. Method: We propose the first open-source, reproducible VisIT data generation framework that automatically constructs high-quality, multi-turn vision-language dialogues from image metadata. Our approach introduces a three-stage open pipeline: (1) metadata clustering and grouping, (2) controllable-precision structured prompt orchestration, and (3) dialogue sampling and resampling optimization—enabling cross-model reproducibility and domain adaptation. The full pipeline is implemented using open models: Gemma 2 27B and LLaMA 3.1 70B. Contribution/Results: Experiments show that, given identical images and metadata, our generated VisIT data improves average data quality by 3% (up to +12% for single tasks), and downstream multimodal large models achieve significant performance gains across multiple benchmarks. The code is publicly released.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Visual Instruction Tuning (VisIT) data, commonly available as human-assistant conversations with images interleaved in the human turns, are currently the most widespread vehicle for aligning strong LLMs to understand visual inputs, converting them to strong LMMs. While many VisIT datasets are available, most are constructed using ad-hoc techniques developed independently by different groups. They are often poorly documented, lack reproducible code, and rely on paid, closed-source model APIs such as GPT-4, Gemini, or Claude to convert image metadata (labels) into VisIT instructions. This leads to high costs and makes it challenging to scale, enhance quality, or generate VisIT data for new datasets. In this work, we address these challenges and propose an open and unified recipe and approach,~ extbf{method}, for converting available metadata to VisIT instructions using open LLMs. Our multi-stage method features an efficient framework for metadata grouping, quality control, data and prompt organization, and conversation sampling. We show that our approach can reproduce or enhance the data quality of available VisIT datasets when applied to the same image data and metadata sources, improving GPT-4 generated VisIT instructions by ~3% on average and up to 12% on individual benchmarks using open models, such as Gemma 2 27B and LLaMa 3.1 70B. Additionally, our approach enables effective performance scaling - both in quantity and quality - by enhancing the resulting LMM performance across a wide range of benchmarks. We also analyze the impact of various factors, including conversation format, base model selection, and resampling strategies. Our code, which supports the reproduction of equal or higher-quality VisIT datasets and facilities future metadata-to-VisIT data conversion for niche domains, is released at https://github.com/jacob-hansen/Instructify.

Problem

• High costs and reliance on closed-source APIs for VisIT data conversion
• Lack of reproducibility and documentation in existing VisIT datasets
• Challenges in scaling and enhancing quality of VisIT instructions

Innovation

• Open unified recipe for metadata conversion
• Multi-stage framework for quality control
• Scaling performance with open LLMs