A Comparison of Sentiment Analysis Tools
Bibliographic record
Abstract
Sentiment analysis (SA), an analytics technique that assesses the “tone” of text, has emerged as a viable alternative to help users decide what to read without analyzing the whole text. In this study, we compare two main SA techniques (lexicon-based and machine-learning) for analyzing sentiments in the context of consumer product reviews. Given that a noted gap in prior research has been the almost sole focus on short textual information that concerns specific contexts (e.g., tweets about the Winter Olympics), we examine the role of contextual factors. Specifically, we examine reviews that address a multitude of product/service contexts, and which vary significantly in length. \\ \\ To test the research model, we collected 625 consumer reviews that ranged in length from 10 to 551 words, and which concerned six goods belonging to the three product/service categories commonly cited in the literature: a) search goods: laptop computers and paper notebooks; b) experience goods: hotels and restaurants; and c) credence goods: car repair and multi-vitamins. To analyze the reviews, we used two tools: 1) VADER (Valence Aware Dictionary for sEntiment Reasoning); a parsimonious rule-based sentiment analysis tool that has been shown to be especially effective in analyzing social media posts, and 2) Google Cloud Natural Language API (henceforth called Google), which uses a machine-learning approach. To assess the effectiveness of the tools, we compared the sentiment scores generated by each tool to the star ratings that were provided by the original authors of the consumer reviews. Hence, the (absolute) difference between the SA scores and the star ratings served as our dependent measure. \\ \\ The results of an ANOVA indicated that the lexicon-based approach (VADER) to sentiment analysis outperforms machine-learning in almost all product contexts regardless of review length (average error M = 16.6% vs. M = 19.8%; F = 13.81, p < 0.01). The main effect of review length was also statistically significant (F = 14.04, p < 0.01), where the SA tools’ accuracy was better for short and medium length reviews. Overall, both tools’ average accuracy was highest for credence goods, followed by experience and then search goods (F = 5.82, p < 0.05). Overall, VADER demonstrated higher accuracy over Google for credence and search goods. Machine learning (Google) had a better accuracy only in the case of long reviews about experience goods (F = 5.64, p < 0.05). \\ \\ The results of this study provide evidence that there are differences in the accuracy of various SA tools, and that it is important to consider contextual factors when choosing a SA tool. Building on these results, we will examine whether SA can be a substitute or a complement to star ratings in consumer decision making. \\
Fetched live from OpenAlex and de-inverted. Abstracts are not stored in this database: the inverted indexes are 8.6 GB of the frame’s 9.3 GB of text, and the host has 13 GB free.
How this classification was reachedexpand
Full frame machine prediction
Teacher imitationNot calibrated prevalence, not ground truth. Human validation pending. The Gemma side is a direct model label for every work in the frame, read from the title-only record. The Codex side is a classifier learned from the 10,348 direct Codex labels and calibrated to design-weighted sample rates; fields without enough sample support carry no Codex call. Candidate is the union of the two sides; consensus is their intersection. These outputs are machine_predicted_unvalidated and are not human labels.
Distilled classifier scores by category (both heads)
| Category | Codex | Gemma |
|---|---|---|
| Metaresearch | 0.009 | 0.030 |
| Meta-epidemiology (narrow) | 0.001 | 0.000 |
| Meta-epidemiology (broad) | 0.001 | 0.002 |
| Bibliometrics | 0.009 | 0.004 |
| Science and technology studies | 0.001 | 0.000 |
| Scholarly communication | 0.003 | 0.004 |
| Open science | 0.001 | 0.002 |
| Research integrity | 0.001 | 0.001 |
| Insufficient payload (model declined to judge) | 0.004 | 0.002 |
Machine scores (provisional)
The two teacher heads of the student model, read on this work. A score orders the frame for review; it never asserts a category, and the validation status ships verbatim with every row.
Baseline scores from an immature model (maturity gate not passed, 7 training rounds). Scores rank; they never assert a category.
score_only:v0-immature-baseline · verbatim from the scoring run: score_only means the number may rank works, and no category label ships from itClassification
machine, unvalidatedMachine predicted; a candidate call from one source (direct Gemma or distilled Codex), not a consensus.
How this classification was reached, model by model and score by score, is at the end of the page under "How this classification was reached".