ChimeraLoRA: Multi-Head LoRA-Guided Synthetic Datasets

2026-02-23

AI Summary

This work addresses model bias and reliability issues arising from insufficient samples of tail classes in data-scarce scenarios by proposing a multi-head LoRA-based diffusion model for synthetic data generation. The approach decouples LoRA adapters into a shared class-prior module (LoRA-A) and a set of image-specific modules (LoRA-B), augmented with class-aware bounding box semantics and Dirichlet mixture sampling. This design simultaneously preserves class-level diversity and instance-level detail fidelity under few-shot conditions. Experimental results demonstrate that the generated images closely align with the true data distribution, significantly improving accuracy and robustness in downstream classification tasks. Notably, this method achieves the first successful co-optimization of semantic coherence and fine-grained detail in few-shot image synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Beyond general recognition tasks, specialized domains including privacy-constrained medical applications and fine-grained settings often encounter data scarcity, especially for tail classes. To obtain less biased and more reliable models under such scarcity, practitioners leverage diffusion models to supplement underrepresented regions of real data. Specifically, recent studies fine-tune pretrained diffusion models with LoRA on few-shot real sets to synthesize additional images. While an image-wise LoRA trained on a single image captures fine-grained details yet offers limited diversity, a class-wise LoRA trained over all shots produces diverse images as it encodes class priors yet tends to overlook fine details. To combine both benefits, we separate the adapter into a class-shared LoRA~$A$ for class priors and per-image LoRAs~$\mathcal{B}$ for image-specific characteristics. To expose coherent class semantics in the shared LoRA~$A$, we propose a semantic boosting by preserving class bounding boxes during training. For generation, we compose $A$ with a mixture of $\mathcal{B}$ using coefficients drawn from a Dirichlet distribution. Across diverse datasets, our synthesized images are both diverse and detail-rich while closely aligning with the few-shot real distribution, yielding robust gains in downstream classification accuracy.

Problem

• data scarcity
• tail classes
• synthetic data generation
• few-shot learning
• domain-specific recognition

Innovation

• ChimeraLoRA
• LoRA
• diffusion models
• few-shot synthesis
• class priors