Best Preprocessing Techniques for Sentiment Analysis

2026-06-22

AI Summary

This study addresses the lack of systematic investigation into the ordering of preprocessing steps in sentiment analysis, which has constrained model performance and efficiency. Focusing on Twitter data, it presents the first quantitative evaluation of the relative impact and optimal sequencing of key preprocessing techniques—including tokenization, text cleaning, stemming, stopword removal (with negations preserved), and spelling correction. Through comprehensive combinatorial experiments, the work demonstrates that tokenization contributes most significantly to model performance, while spelling correction has the least effect. The identified optimal pipeline—tokenization followed by text cleaning, stemming, and stopword removal—substantially enhances model effectiveness and reduces trial-and-error costs, establishing a reproducible and efficient preprocessing paradigm for sentiment analysis.

Abstract

Sentiment analysis in Twitter datasets is important because it enables monitoring public opinion on products and analysis of political and social movements. One critical step is preprocessing: the automated processing of text for machine learning algorithms. Preprocessing plays a critical role in reducing noise and improving efficiency. However, little research has systematically examined the order in which preprocessing techniques are implemented. We find that, when accounting for order, spelling correction is the least impactful preprocessing technique, whereas tokenisation is the most impactful. Stemming and stop-word removal are interchangeable, and it is better to remove stop words without removing negation. The best order for applying the preprocessing techniques was tokenisation, text cleaning, stemming, and then stopword removal. Our results provide a systematic approach for practitioners to deploy preprocessing to improve model output without the costly preprocessing exploratory phase.

Problem

• sentiment analysis
• preprocessing
• Twitter datasets
• text processing
• machine learning

Innovation

• preprocessing order
• sentiment analysis
• tokenisation
• stop-word removal
• stemming