LTDA-Drive: LLMs-guided Generative Models based Long-tail Data Augmentation for Autonomous Driving

2025-05-21

AI Summary

To address the poor detection performance of safety-critical tail classes (e.g., pedestrians, cyclists) in autonomous driving 3D perception—caused by long-tailed class distribution—this paper proposes the first LLM-guided generative long-tail data augmentation framework. Methodologically, it introduces an LLM-based intelligent filtering mechanism and a three-stage text-guided diffusion synthesis pipeline, integrating KITTI scene structural priors, a generative object insertion module, and an LLM-powered quality evaluation agent to enable controllable, high-fidelity, and diverse synthesis of tail-class samples. Unlike conventional re-sampling or re-weighting approaches, our framework overcomes fundamental trade-offs between diversity and fidelity. On the KITTI benchmark, it achieves a 34.75% improvement in rare-class detection performance, significantly enhancing model robustness for safety-critical tail classes.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

3D perception plays an essential role for improving the safety and performance of autonomous driving. Yet, existing models trained on real-world datasets, which naturally exhibit long-tail distributions, tend to underperform on rare and safety-critical, vulnerable classes, such as pedestrians and cyclists. Existing studies on reweighting and resampling techniques struggle with the scarcity and limited diversity within tail classes. To address these limitations, we introduce LTDA-Drive, a novel LLM-guided data augmentation framework designed to synthesize diverse, high-quality long-tail samples. LTDA-Drive replaces head-class objects in driving scenes with tail-class objects through a three-stage process: (1) text-guided diffusion models remove head-class objects, (2) generative models insert instances of the tail classes, and (3) an LLM agent filters out low-quality synthesized images. Experiments conducted on the KITTI dataset show that LTDA-Drive significantly improves tail-class detection, achieving 34.75% improvement for rare classes over counterpart methods. These results further highlight the effectiveness of LTDA-Drive in tackling long-tail challenges by generating high-quality and diverse data.

Problem

• Improving 3D perception for rare classes in autonomous driving
• Addressing long-tail data scarcity in real-world datasets
• Enhancing detection of vulnerable road users like pedestrians

Innovation

• LLM-guided text-to-image synthesis
• Three-stage object replacement process
• Quality filtering via LLM agent