SemiScope: Disentangling Classifier Tuning and Joint Optimization in Semi-Supervised Security Classification

2026-06-30

AI Summary

This work addresses the scarcity of labeled data in security classification and the limitations of existing semi-supervised methods, which often employ black-box designs and fixed configurations that struggle with class imbalance induced by pseudo-labeling. To overcome these challenges, the authors propose SemiScope, an analysis framework that jointly optimizes self-training pipelines, confidence-based filtering, oversampling strategies, and tree-based classifiers via Bayesian optimization. A controlled experimental protocol is introduced to disentangle the contributions of classifier hyperparameter tuning from end-to-end optimization. Evaluations across five security datasets demonstrate that SemiScope substantially outperforms default semi-supervised baselines; notably, optimizing only the classifier’s hyperparameters and decision threshold recovers a median of 86% of the full performance gain, achieving near fully supervised accuracy at labeling rates of 20–40%. The study also establishes a reusable decomposition protocol that highlights the pivotal role of classifier tuning.

Abstract

Background. Labeled data for security classification is scarce. Semi-supervised learning (SSL) propagates labels from a small labeled pool to larger unlabeled pools. Yet security applications often use SSL as a black box: default parameters, a fixed classifier, and no handling of pseudo-label-induced class imbalance. Aims. Recent work reports sizeable gains from optimizing SSL pipelines via joint search, AutoML, or per-component tuning. These gains are hard to attribute: they may reflect useful SSL-classifier interactions, or mostly from simply tuning the downstream classifier. We disentangle these effects for binary tabular security data with classical SSL and tree-based classifiers. Method. We build SemiScope as an analysis instrument, not a deployment recommendation. It uses Bayesian Optimization to jointly tune SSL settings, confidence filtering, oversampling, and the classifier. The key control, Tuned-Clf, fixes SSL to defaults but gets the same 100-trial classifier budget and validation-set threshold tuning as SemiScope. At 10% labels, we compare them with paired TOST using a +/-1.0 g-measure smallest effect of interest. Results. SemiScope beats every default SSL baseline on all five datasets, improving over the strongest by 0.7-12.7 points. Under the equal-budget control, Tuned-Clf is statistically equivalent to the full pipeline on 4 of 5 datasets; Phishing is inconclusive. Classifier HPO alone recovers a median 86% of SemiScope's gain over Default Self-Training (ST) + Random Forest (RF). Conclusions. The reusable contribution is the decomposition protocol. A simpler recipe suffices: use Self-Training, tune the classifier with Bayesian Optimization, and tune the decision threshold on validation data. It reaches within 1 g-measure of Supervised RF at 20-30% labels on four datasets and 40% on Drebin, at the same or lower label rate than Default ST + RF on every dataset.

Problem

• semi-supervised learning
• security classification
• classifier tuning
• pseudo-labeling
• class imbalance

Innovation

• semi-supervised learning
• classifier tuning
• Bayesian optimization
• disentanglement analysis
• security classification