RoBERTa-BiLSTM: A Context-Aware Hybrid Model for Sentiment Analysis

2024-06-01

arXiv.org

AI Summary

To address challenges in sentiment analysis—including lexical diversity, long-range dependencies, out-of-vocabulary words, and class imbalance—this paper proposes a deeply coupled RoBERTa-BiLSTM hybrid model. RoBERTa serves as the semantic encoder to extract rich contextualized representations, while BiLSTM captures fine-grained sequential dependencies, synergistically combining Transformer-based parallelism with recurrent modeling of long-distance contextual relationships. The model is trained end-to-end with five-fold cross-validation and evaluated jointly on three benchmark datasets: Twitter US Airline, IMDb, and Sentiment140. Experimental results demonstrate state-of-the-art accuracy of 80.74%, 92.36%, and 82.25%, respectively—significantly outperforming BERT and RoBERTa-base baselines. This work pioneers deep integration of pretrained semantic modeling with dynamic sequence modeling, establishing a novel paradigm for lightweight yet high-performance sentiment analysis.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Effectively analyzing the comments to uncover latent intentions holds immense value in making strategic decisions across various domains. However, several challenges hinder the process of sentiment analysis including the lexical diversity exhibited in comments, the presence of long dependencies within the text, encountering unknown symbols and words, and dealing with imbalanced datasets. Moreover, existing sentiment analysis tasks mostly leveraged sequential models to encode the long dependent texts and it requires longer execution time as it processes the text sequentially. In contrast, the Transformer requires less execution time due to its parallel processing nature. In this work, we introduce a novel hybrid deep learning model, RoBERTa-BiLSTM, which combines the Robustly Optimized BERT Pretraining Approach (RoBERTa) with Bidirectional Long Short-Term Memory (BiLSTM) networks. RoBERTa is utilized to generate meaningful word embedding vectors, while BiLSTM effectively captures the contextual semantics of long-dependent texts. The RoBERTa-BiLSTM hybrid model leverages the strengths of both sequential and Transformer models to enhance performance in sentiment analysis. We conducted experiments using datasets from IMDb, Twitter US Airline, and Sentiment140 to evaluate the proposed model against existing state-of-the-art methods. Our experimental findings demonstrate that the RoBERTa-BiLSTM model surpasses baseline models (e.g., BERT, RoBERTa-base, RoBERTa-GRU, and RoBERTa-LSTM), achieving accuracies of 80.74%, 92.36%, and 82.25% on the Twitter US Airline, IMDb, and Sentiment140 datasets, respectively. Additionally, the model achieves F1-scores of 80.73%, 92.35%, and 82.25% on the same datasets, respectively.

Problem

• Addresses lexical diversity and long dependencies in sentiment analysis
• Overcomes challenges of unknown symbols and imbalanced datasets
• Improves execution time and accuracy in sentiment analysis tasks

Innovation

• Combines RoBERTa and BiLSTM for sentiment analysis
• Uses RoBERTa for meaningful word embeddings
• BiLSTM captures contextual semantics in texts