Fault of Our Stars: Behavioral Drivers of Rating-Sentiment Incongruence

2026-06-24

AI Summary

This study challenges the common practice of treating star ratings as direct weak labels for textual sentiment by investigating discrepancies between assigned ratings and expressed sentiment in online reviews. Analyzing 16,156 reviews of Sri Lankan tourist attractions, the authors employ a rating-agnostic Transformer-based sentiment analysis pipeline combined with logistic regression, random forest models, and SHAP interpretability techniques to systematically identify six directional inconsistency patterns for the first time. They propose behavioral mechanisms such as “conservative raters” and “obligatory five-star reviewers” to explain these mismatches. The findings reveal that 18.6% of reviews exhibit sentiment-rating inconsistency—highest among museum-type venues—with the degree of inconsistency significantly influenced by venue category, user experience level, review length, and temporal factors.

Abstract

When people share experiences online, they often express thoughts in two ways: a star rating and a written review. In sentiment analysis, ratings are widely used as convenient weak labels for textual sentiment, yet whether the two actually agree is rarely questioned. This study investigates sentiment-rating incongruence, where the sentiment expressed in review text differs from the sentiment implied by the assigned star rating, in Sri Lankan tourism attraction reviews. A dataset of 16,156 reviews from 2010 to 2023 is analyzed using a transformer-based sentiment pipeline that derives textual sentiment independently of assigned ratings. Incongruence occurs in 18.6% of reviews and falls into six directional patterns, with Conservative Rater and Obligatory 5-Star behaviors accounting for the majority of mismatches. Prevalence also varies across venue types, with museums showing the highest rates. Statistical tests, logistic regression, Random Forest, and SHAP analysis identify venue type, reviewer expertise, review length, and temporal factors as contributors to rating-text divergence. Overall, this study demonstrates that star ratings are not interchangeable with textual sentiment and should be validated before being treated as ground-truth labels in NLP.

Problem

• sentiment-rating incongruence
• star ratings
• textual sentiment
• online reviews
• weak labels

Innovation

• sentiment-rating incongruence
• weak labels
• transformer-based sentiment analysis
• behavioral patterns
• review authenticity