Beyond Visual Similarity: Rule-Guided Multimodal Clustering with explicit domain rules

2025-09-24

AI Summary

Traditional clustering methods rely solely on data similarity, making it difficult to incorporate domain-specific structural or semantic constraints—thus limiting interpretability and practical utility. To address this, we propose DARTVAE, the first multimodal clustering framework that explicitly integrates domain rules as primary learning signals. Leveraging large language models (LLMs), we automatically generate domain rules and construct a knowledge graph; we then design rule-consistency and rule-violation penalty terms embedded within the variational autoencoder (VAE) objective function, enabling structure-aware joint representation learning in the latent space. Our approach unifies multimodal feature learning, rule-guided optimization, and interpretable modeling. Evaluated on aviation and automotive datasets, DARTVAE significantly improves clustering plausibility—e.g., correctly separating drones, grouping stealth aircraft, and distinguishing SUVs from sedans—aligning with expert domain knowledge while outperforming state-of-the-art baselines on standard metrics including ARI and NMI.

Technology Category

• Machine Learning: Clustering
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Knowledge Acquisition

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Traditional clustering techniques often rely solely on similarity in the input data, limiting their ability to capture structural or semantic constraints that are critical in many domains. We introduce the Domain Aware Rule Triggered Variational Autoencoder (DARTVAE), a rule guided multimodal clustering framework that incorporates domain specific constraints directly into the representation learning process. DARTVAE extends the VAE architecture by embedding explicit rules, semantic representations, and data driven features into a unified latent space, while enforcing constraint compliance through rule consistency and violation penalties in the loss function. Unlike conventional clustering methods that rely only on visual similarity or apply rules as post hoc filters, DARTVAE treats rules as first class learning signals. The rules are generated by LLMs, structured into knowledge graphs, and enforced through a loss function combining reconstruction, KL divergence, consistency, and violation penalties. Experiments on aircraft and automotive datasets demonstrate that rule guided clustering produces more operationally meaningful and interpretable clusters for example, isolating UAVs, unifying stealth aircraft, or separating SUVs from sedans while improving traditional clustering metrics. However, the framework faces challenges: LLM generated rules may hallucinate or conflict, excessive rules risk overfitting, and scaling to complex domains increases computational and consistency difficulties. By combining rule encodings with learned representations, DARTVAE achieves more meaningful and consistent clustering outcomes than purely data driven models, highlighting the utility of constraint guided multimodal clustering for complex, knowledge intensive settings.

Problem

• Incorporating domain rules into clustering beyond visual similarity
• Enforcing semantic constraints through rule-guided representation learning
• Addressing limitations of traditional clustering with multimodal constraints

Innovation

• Rule-guided VAE clustering with domain constraints
• LLM-generated rules enforced through loss penalties
• Multimodal latent space combining semantic and visual features