Augmented Relevance Datasets with Fine-Tuned Small LLMs
Manual annotation of query-document relevance is costly and poorly scalable. Method: This paper proposes an automatic relevance labeling and data augmentation framework based on lightweight fine-tuning of small-scale large language models (LLMs). It systematically employs parameter-efficient fine-tuning (PEFT) of compact LLMs for relevance modeling—replacing both manual annotation and expensive black-box models—while preserving annotation accuracy and substantially reducing computational and human costs. The framework comprises four components: LLM fine-tuning, fine-grained relevance assessment, synthetic data generation, and joint optimization with downstream ranking models. Results: The fine-tuned small LLM outperforms several proprietary large models in relevance judgment. When used to augment training data, it improves the downstream ranking model’s NDCG@10 by up to 12.7%, demonstrating the effectiveness and practicality of this efficient, high-quality data augmentation paradigm.