🤖 AI Summary
This work addresses the key challenge in multimodal sarcasm and cyberbullying detection—modeling multi-level semantic incongruities between textual and visual modalities. To this end, the authors propose the HCIG framework, which introduces, for the first time, a hierarchical cross-modal inconsistency graph network. This architecture leverages graph attention mechanisms to capture fine-grained to holistic semantic contradictions across token, phrase, and global levels, and employs hierarchical attention to adaptively fuse multi-granularity representations. Additionally, a Graph-based Contradiction-aware Convolutional Network (GCCN) is designed with contradiction-aware graph pooling to enable efficient cross-modal reasoning. Experimental results demonstrate that HCIG achieves state-of-the-art performance, attaining 85.74% accuracy and 85.29% macro F1 on the MMSD dataset, and 69.62% accuracy with 74.90% bullying-class F1 on the MultiBully dataset.
📝 Abstract
Multimodal sarcasm and cyberbullying detection remain challenging because the intended meaning often emerges from incongruity between textual and visual information rather than from either modality alone. Existing multimodal approaches primarily rely on feature fusion or cross-modal attention, which may not effectively capture hierarchical semantic inconsistencies across different levels of representation. To address this limitation, this paper proposes HCIG (Hierarchical Cross-modal Incongruity Graph Network), a novel framework that models cross-modal incongruity at token, phrase, and global levels using graph attention networks and adaptively integrates these representations through a learned hierarchical attention mechanism. As a complementary architecture, we also introduce GCCN (Graph-based Cross-modal Contradiction Network), which performs graph-based reasoning using contradiction-aware pooling for efficient multimodal interaction learning. The proposed models are evaluated on the MMSD sarcasm benchmark and the MultiBully cyberbullying dataset, together with comprehensive ablation studies and cross-task transfer experiments. Experimental results demonstrate that HCIG achieves the best performance on MMSD with 85.74% accuracy and 85.29% macro-F1, while GCCN attains the highest macro-F1 (68.66%) on MultiBully and HCIG achieves the highest accuracy (69.62%) and bullying-class F1 (74.90%). The findings demonstrate that hierarchical multi-granularity incongruity modeling provides more effective multimodal reasoning than conventional fusion strategies, offering a robust framework for sarcasm and cyberbullying detection in social media.