From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs

2026-01-07

arXiv.org

AI Summary

This work addresses the limitation of existing large language models (LLMs) in data augmentation, which typically rely on heuristic or brute-force search strategies without awareness of downstream task performance. The authors propose the first performance-aware closed-loop system that enables LLMs to internalize semantic-level performance signals—guided solely by empirical performance rankings without reinforcement learning or explicit reward models—and autonomously design task-aligned data transformation strategies. Leveraging a curated dataset of over 6,000 PyTorch augmentation functions annotated with downstream accuracy, the system aligns LLM outputs via LoRA fine-tuning and pairwise performance ranking. Compared to brute-force search, the approach reduces candidate evaluations by up to 600× while maintaining competitive peak accuracy, demonstrating that the model learns semantic performance cues rather than memorizing syntactic patterns.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches. We introduce a performance-aware, closed-loop solution in the NNGPT ecosystem of projects that enables LLMs to autonomously engineer optimal transformations by internalizing empirical performance cues. We fine-tune LLMs with Low-Rank Adaptation on a novel repository of more than 6,000 empirically evaluated PyTorch augmentation functions, each annotated solely by downstream model accuracy. Training uses pairwise performance ordering (better-worse transformations), enabling alignment through empirical feedback without reinforcement learning, reward models, or symbolic objectives. This reduces the need for exhaustive search, achieving up to 600x times fewer evaluated candidates than brute-force discovery while maintaining competitive peak accuracy and shifting generation from random synthesis to task-aligned design. Ablation studies show that structured Chain-of-Thought prompting introduces syntactic noise and degrades performance, whereas direct prompting ensures stable optimization in performance-critical code tasks. Qualitative and quantitative analyses demonstrate that the model internalizes semantic performance cues rather than memorizing syntax. These results show that LLMs can exhibit task-level reasoning through non-textual feedback loops, bypassing explicit symbolic rewards.

Problem

• data augmentation
• large language models
• code synthesis
• performance-guided design
• empirical feedback

Innovation

• performance-guided transformation
• LLM fine-tuning
• data augmentation
• empirical feedback loop
• Low-Rank Adaptation