Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure

2026-07-13

AI Summary

This work addresses the dual challenge of extreme class imbalance and strong hierarchical dependencies in the Common Weakness Enumeration (CWE) classification for cybersecurity vulnerabilities. The authors propose a hierarchy-aware RoBERTa framework that explicitly models the CWE taxonomy by incorporating learnable parent-class embeddings, thereby enforcing taxonomic consistency in predictions. The study reveals, for the first time, that conventional oversampling techniques such as SMOTE disrupt parent-child category constraints in high-dimensional embedding spaces, and demonstrates that hierarchy-aware representation learning outperforms data augmentation strategies. Evaluated on the CWE Research Concepts dataset, the proposed method achieves a weighted F1 score of 0.76 without any data augmentation, while substantially improving the minority-class F1 score from 0.40 to 0.60.

Abstract

Classifying cybersecurity vulnerabilities using the Common Weakness Enumeration (CWE) taxonomy is challenging due to extreme class imbalance and strong hierarchical dependencies among weakness categories. Although oversampling techniques such as Synthetic Minority Oversampling Technique (SMOTE) and Adaptive Synthetic Sampling (ADASYN) are widely adopted to mitigate class imbalance, their effectiveness for hierarchical CWE text classification remains largely unexplored. This paper proposes a Hierarchy-Aware RoBERTa framework that explicitly incorporates CWE structural information through learnable parent-class embeddings, preserving taxonomic consistency. Our experiments demonstrate that synthetic interpolation in high-dimensional embedding spaces violates the inherent parent-child constraints of the CWE hierarchy, offering only marginal benefits for classical ML models while consistently degrading deep learning architectures. Evaluated on a CWE Research Concept dataset, the proposed model achieves a weighted F1-score of 0.76 without data augmentation, outperforming all baselines with notable gains on minority classes, including the Class category whose F1-score improved from 0.40 to 0.60 over the BERT baseline. Our results suggest that hierarchy-aware representation learning is a more principled alternative to oversampling for structured vulnerability classification.

Problem

• class imbalance
• hierarchical dependencies
• CWE taxonomy
• vulnerability classification
• text classification

Innovation

• Hierarchy-Aware Representation Learning
• Class Imbalance
• CWE Taxonomy
• RoBERTa
• Structured Text Classification