Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation

2026-06-16

AI Summary

This work addresses the challenges of high inference costs and limited diversity in synthetic data generation for low-resource languages, which often rely on few-shot prompting. The authors propose an activation steering approach that manipulates activations in early layers of large language models to perform language identity and text quality interventions under both zero-shot and few-shot settings—without requiring any target-language examples. This method effectively guides the model to produce higher-quality and more diverse synthetic data. Experiments across eleven typologically diverse low-resource languages demonstrate that the proposed technique significantly enhances output diversity and consistently improves downstream classification performance after fine-tuning.

Abstract

Large language models (LLMs) have become an effective tool for synthetic data generation, including for low-resource languages, where generated data can improve downstream task performance. Current best-performing approaches typically rely on few-shot prompting with target-language examples, which increases inference costs and may reduce diversity through lexical anchoring. In this work, we investigate activation steering as an alternative for low-resource synthetic data generation. We study two steering strategies: Language Steering, which targets the linguistic identity of a language, and Quality Steering, which captures well-formedness by contrasting human-written and backtranslated text representations. We evaluate these methods across four open-source LLMs, multiple layers, and 11 typologically diverse languages by generating sentiment and topic classification data and finetuning smaller classifiers. Steering is applied in both zero-shot and few-shot prompting settings and compared against non-steered counterparts. Our results show that steering on early layers consistently improves the diversity of generated data while often yielding stronger downstream model performance, particularly for low-resource languages.

Problem

• synthetic data
• low-resource languages
• data diversity
• language generation
• lexical anchoring

Innovation

• activation steering
• synthetic data generation
• low-resource languages
• language steering
• quality steering