Detecting Malicious Concepts Without Image Generation in AIGC

2025-02-13

AI Summary

Malicious concept camouflage uploads pose a content pollution risk in AIGC concept-sharing platforms. Method: This paper proposes Concept QuickLook—the first image-generation-free, file-level malicious concept detection method. It introduces the first systematic formalization of “malicious concepts” and establishes a multi-granularity detection framework integrating concept embedding representation analysis, semantic consistency verification, and robustness-oriented adversarial testing, supporting both exact matching and fuzzy identification modes. Contribution/Results: Unlike existing generation-dependent verification approaches, Concept QuickLook eliminates risks of spurious generation and reduces computational overhead. It achieves >98% detection accuracy across multiple real-world concept datasets, accelerates inference by over two orders of magnitude, and has been successfully deployed and validated on an operational sharing platform.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The task of text-to-image generation has achieved tremendous success in practice, with emerging concept generation models capable of producing highly personalized and customized content. Fervor for concept generation is increasing rapidly among users, and platforms for concept sharing have sprung up. The concept owners may upload malicious concepts and disguise them with non-malicious text descriptions and example images to deceive users into downloading and generating malicious content. The platform needs a quick method to determine whether a concept is malicious to prevent the spread of malicious concepts. However, simply relying on concept image generation to judge whether a concept is malicious requires time and computational resources. Especially, as the number of concepts uploaded and downloaded on the platform continues to increase, this approach becomes impractical and poses a risk of generating malicious content. In this paper, we propose Concept QuickLook, the first systematic work to incorporate malicious concept detection into research, which performs detection based solely on concept files without generating any images. We define malicious concepts and design two work modes for detection: concept matching and fuzzy detection. Extensive experiments demonstrate that the proposed Concept QuickLook can detect malicious concepts and demonstrate practicality in concept sharing platforms. We also design robustness experiments to further validate the effectiveness of the solution. We hope this work can initiate malicious concept detection tasks and provide some inspiration.

Problem

• Detect malicious concepts without image generation
• Prevent malicious content spread on platforms
• Quickly assess concept safety using concept files

Innovation

• Malicious concept detection
• No image generation
• Concept matching and fuzzy detection