Are Large Language Models Good Data Preprocessors?

2025-02-24

AI Summary

This study investigates the efficacy of large language models (LLMs) as caption denoisers for image descriptions—specifically, noisy captions generated by BLIP/GIT—in multimodal tasks such as meme persuasion detection. Method: Moving beyond prior work limited to simple datasets and few models (e.g., GPT), we conduct the first systematic evaluation of state-of-the-art LLMs—including LLaMA-3.1-70B, GPT-4 Turbo, and Sonnet-3.5-v2—in realistic, complex scenarios. We integrate prompt engineering with end-to-end validation on SemEval 2024’s multilabel downstream task and assess statistical significance via t-tests and Wilcoxon signed-rank tests. Contribution/Results: LLM-based caption cleaning yields only marginal performance gains; most improvements are statistically insignificant. Our findings reveal that denoising effectiveness is highly contingent on task complexity and noise characteristics, thereby challenging the implicit assumption that LLMs serve as universally effective caption cleaners.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

High-quality textual training data is essential for the success of multimodal data processing tasks, yet outputs from image captioning models like BLIP and GIT often contain errors and anomalies that are difficult to rectify using rule-based methods. While recent work addressing this issue has predominantly focused on using GPT models for data preprocessing on relatively simple public datasets, there is a need to explore a broader range of Large Language Models (LLMs) and tackle more challenging and diverse datasets. In this study, we investigate the use of multiple LLMs, including LLaMA 3.1 70B, GPT-4 Turbo, and Sonnet 3.5 v2, to refine and clean the textual outputs of BLIP and GIT. We assess the impact of LLM-assisted data cleaning by comparing downstream-task (SemEval 2024 Subtask"Multilabel Persuasion Detection in Memes") models trained on cleaned versus non-cleaned data. While our experimental results show improvements when using LLM-cleaned captions, statistical tests reveal that most of these improvements are not significant. This suggests that while LLMs have the potential to enhance data cleaning and repairing, their effectiveness may be limited depending on the context they are applied to, the complexity of the task, and the level of noise in the text. Our findings highlight the need for further research into the capabilities and limitations of LLMs in data preprocessing pipelines, especially when dealing with challenging datasets, contributing empirical evidence to the ongoing discussion about integrating LLMs into data preprocessing pipelines.

Problem

• Evaluate LLMs as data preprocessors
• Clean BLIP and GIT textual outputs
• Assess LLM impact on downstream tasks

Innovation

• LLMs for text cleaning
• Multimodal data refinement
• Empirical LLM effectiveness analysis