SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model

2024-06-17

arXiv.org

AI Summary

Current vision-language models (VLMs) lack high-quality, multimodal preference data for safety alignment. Method: We introduce SPA-VL—the first large-scale, multidimensional safety preference alignment dataset for VLMs—comprising over 100K quadruple samples spanning six harm domains and 53 fine-grained subcategories. We propose a novel, fine-grained VLM safety preference taxonomy and an end-to-end automated pipeline for preference data construction, integrating responses from 12 open- and closed-source VLMs to ensure diversity. Alignment training leverages multi-model response sampling, automatic preference annotation, cross-domain harm modeling, and a joint safety-usefulness evaluation framework. Contribution/Results: Aligned VLMs achieve substantial gains in harm avoidance (+28.6%) and response usefulness (+21.3%), without degrading core vision-language understanding capabilities.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

The emergence of Vision Language Models (VLMs) has brought unprecedented advances in understanding multimodal information. The combination of textual and visual semantics in VLMs is highly complex and diverse, making the safety alignment of these models challenging. Furthermore, due to the limited study on the safety alignment of VLMs, there is a lack of large-scale, high-quality datasets. To address these limitations, we propose a Safety Preference Alignment dataset for Vision Language Models named SPA-VL. In terms of breadth, SPA-VL covers 6 harmfulness domains, 13 categories, and 53 subcategories, and contains 100,788 samples of the quadruple (question, image, chosen response, rejected response). In terms of depth, the responses are collected from 12 open-source (e.g., QwenVL) and closed-source (e.g., Gemini) VLMs to ensure diversity. The construction of preference data is fully automated, and the experimental results indicate that models trained with alignment techniques on the SPA-VL dataset exhibit substantial improvements in harmlessness and helpfulness while maintaining core capabilities. SPA-VL, as a large-scale, high-quality, and diverse dataset, represents a significant milestone in ensuring that VLMs achieve both harmlessness and helpfulness.

Problem

• Addresses safety alignment in Vision Language Models
• Lacks large-scale, high-quality datasets for VLMs
• Ensures VLMs achieve harmlessness and helpfulness

Innovation

• Automated preference data construction
• Diverse harmfulness domain coverage
• Safety alignment technique enhancement