Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection

2026-05-27

AI Summary

This work addresses the limited generalization of safety detection models due to a scarcity of training examples that violate the HHH (Helpful, Harmless, Honest) principles. The authors explore Activation Steering to generate high-quality synthetic data and introduce, for the first time, an evaluation framework incorporating both sample-level and set-level diversity. Their analysis reveals a negative correlation between diversity and steering intensity. They propose using the harmonic mean of success rate, coherence, and diversity to predict downstream classifier performance. Across experiments combining four concepts, two language models, and four steering methods, Activation Steering outperforms prompt-based generation on three concepts; however, only 41 out of 136 configurations achieve superior results, highlighting the necessity of carefully balancing success, coherence, and diversity to optimize overall effectiveness.

Abstract

Safety detection models require examples of HHH (Helpful, Harmless, Honest)-violating outputs for robust generalization, however such examples are scarce. Activation Steering (AS) has emerged as a data-efficient method for generating target-concept-aligned responses. We investigate whether AS can generate high-quality training datasets for downstream classifiers, a question that remains untested. We present a two-fold study with intrinsic and extrinsic evaluation across $4$ concepts $\times\,2$ models $\times\,4$ steering methods. Intrinsically, beyond the field-standard rubric of steering success (concept alignment) and coherence, we introduce sample- and set-level diversity as a quality axis previously absent from the literature, and find that increasing steering strength reduces response diversity. Extrinsically, we replace HHH-violating examples in the available training data with steered generations and fine-tune detection classifiers. AS-generated data results in a better classifier than the prompting-generated data on $3$ of $4$ concepts. However, only $41$ of $136$ AS configurations outperform prompting, indicating that downstream utility lies in a narrow regime that jointly satisfies success, coherence, and diversity. The harmonic mean of these three axes correlates with downstream AUROC more consistently across concepts than success and coherence alone, providing a practical heuristic target for practitioners tuning AS hyperparameters. Together, our results highlight the potential of AS in synthetic data generation for improving safety detection and identify diversity as a critical, previously overlooked axis for tuning AS.

Problem

• Safety Detection
• Synthetic Data Generation
• Activation Steering
• HHH Violations
• Diversity

Innovation

• Activation Steering
• Synthetic Data Generation
• Diversity
• Safety Detection
• HHH Violation