Evaluating the Effectiveness of Text Pre-Processing in Sentiment Analysis
Source abstract
Abstract
Practical demands and academic challenges have both contributed to making sentiment analysis a thriving area of research. Given that a great deal of sentiment analysis work is performed on social media communications, where text frequently ignores the rules of grammar and spelling, pre-processing techniques are required to clean the data. Pre-processing is also required to normalise the text before undertaking the analysis, as social media is inundated with abbreviations, emoticons, emojis, truncated sentences, and slang. While pre-processing has been widely discussed in the literature, and it is considered indispensable, recommendations for best practice have not been conclusive. Thus, we have reviewed the available research on the subject and evaluated various combinations of pre-processing components quantitatively. We have focused on the case of Twitter sentiment analysis, as Twitter has proved to be an important source of publicly accessible data. We have also assessed the effectiveness of different combinations of pre-processing components for the overall accuracy of a couple of off-the-shelf tools and one algorithm implemented by us. Our results confirm that the order of the pre-processing components matters and significantly improves the performance of naïve Bayes classifiers. We also confirm that lemmatisation is useful for enhancing the performance of an index, but it does not notably improve the quality of sentiment analysis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Advanced Text Analysis Techniques · Topic Modeling
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Engineering, Computing and Mathematics (SECaM), University of Plymouth, Plymouth PL4 8AA, UK
Funding
Interreg 2 Seas Mers Zeeën AGE IN project
References (79)
Show source-provided references
1. Sentiment Analysis and Subjectivity
2. Wiebe, J.M. (1990). Recognizing Subjective Sentences: A Computational Investigation of Narrative Text. [Ph.D. Thesis, State University of New York at Buffalo].
3. Wiebe, J.M., Bruce, R.F., and O’Hara, T.P. (1999, January 20–26). Development and Use of a Gold-Standard Data Set for Subjectivity Classifications. Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, College Park, MD, USA.10.3115/1034678.1034721 ↗
4. Dave, K., Lawrence, S., and Pennock, D.M. (2003, January 20–24). Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews. Proceedings of the 12th International Conference on World Wide Web, Budapest, Hungary.10.1145/775152.775226 ↗
5. Pak, A., and Paroubek, P. (2010, January 17–23). Twitter as a Corpus for Sentiment Analysis and Opinion Mining. Proceedings of the International Conference on Language Resources and Evaluation (LREC), Valletta, Malta.
6. Solangi, Y.A., Solangi, Z.A., Aarain, S., Abro, A., Mallah, G.A., and Shah, A. (2018, January 22–23). Review on Natural Language Processing (NLP) and Its Toolkits for Opinion Mining and Sentiment Analysis. Proceedings of the 5th International Conference on Engineering Technologies and Applied Sciences (ICETAS), Bangkok, Thailand.10.1109/ICETAS.2018.8629198 ↗
7. Gaining Competitive Intelligence from Social Media Data: Evidence from Two Largest Retail Chains in the World10.1108/IMDS-03-2015-0098 ↗
8. Nguyen, T.H., and Shirai, K. (2015, January 26–31). Topic Modeling Based Sentiment Analysis on Social Media for Stock Market Prediction. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China.10.3115/v1/P15-1131 ↗
9. Wang, H., Can, D., Kazemzadeh, A., Bar, F., and Narayanan, S. (2012, January 12–13). A System for Real-Time Twitter Sentiment Analysis of 2012 U.S. Presidential Election Cycle. Proceedings of the ACL 2012 System Demonstrations, ACL ’12, Jeju Island, Korea.
10. Palomino, M., Taylor, T., Göker, A., Isaacs, J., and Warber, S. (2016). The Online Dissemination of Nature–Health Concepts: Lessons from Sentiment Analysis of Social Media Relating to Nature-Deficit Disorder. Int. J. Environ. Res. Public Health, 13.10.3390/ijerph13010142 ↗
11. Twitter Mining for Fine-Grained Syndromic Surveillance10.1016/j.artmed.2014.01.002 ↗
12. The Evolution of Sentiment Analysis—A Review of Research Topics, Venues, and Top Cited Papers10.1016/j.cosrev.2017.10.002 ↗
13. Tokenization10.1007/978-94-015-9273-4_9 ↗
14. Saif, H., Fernandez, M., He, Y., and Alani, H. (2014, January 26–31). On Stopwords, Filtering and Data Sparsity for Sentiment Analysis of Twitter. Proceedings of the International Conference on Language Resources and Evaluation (LREC), Reykjavik, Iceland.
15. Belew, R.K. (2000). Finding Out about: A Cognitive Perspective on Search Engine Technology and the WWW, Cambridge University Press.
16. A comparative study of stemming algorithms
17. Gesmundo, A., and Samardzic, T. (2012, January 8–14). Lemmatisation as a Tagging Task. Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Jeju Island, Korea.
18. Lyu, C., Zhang, Y., and Ji, D. (2016, January 12–17). Joint Word Segmentation, Pos-Tagging and Syntactic Chunking. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AR, USA.10.1609/aaai.v30i1.10369 ↗
19. Murthy, D. (2018). Twitter, Polity Press Cambridge.
20. Kaggle Inc (2022, July 29). Kaggle: Your Machine Learning and Data Science Community. Available online: https://www.kaggle.com/.
21. Lowd, D., and Domingos, P. (2005, January 7–11). Naive Bayes Models for Probability Estimation. Proceedings of the International Conference on Machine Learning, ICML ’05, Bonn, Germany.10.1145/1102351.1102418 ↗
22. Scopus Database: A Review10.1186/1742-5581-3-1 ↗
23. Burgess, J., and Green, J. (2018). YouTube: Online Video and Participatory Culture, John Wiley & Sons.
24. Facebook: A Literature Review10.1177/1461444813488061 ↗
25. Tankovska, H. (2022, July 29). Social Media Usage in the United Kingdom (UK)—Statistics & Facts. Available online: https://cybercrew.uk/blog/social-media-statistics-uk/.
26. Tankovska, H. (2021). Leading Social Networks by Share of Website Visits in the United Kingdom (UK) as of January 2021, Statista Inc.
27. Carnell, J., Linwood, J., and Zawadzki, M. (2003). Creating a Search Engine with Lucene. Professional Struts Applications: Building Web Sites with Struts, ObjectRelationalBridge, Lucene, and Velocity, Apress.10.1007/978-1-4302-1122-8 ↗
28. Bastian, M., Heymann, S., and Jacomy, M. (2009, January 17–20). Gephi: An Open Source Software for Exploring and Manipulating Networks. Proceedings of the International AAAI Conference On web and Social Media, San Jose, CA, USA.10.1609/icwsm.v3i1.13937 ↗
29. A Review of Natural Language Processing Techniques for Opinion Mining Systems10.1016/j.inffus.2016.10.004 ↗
30. Comparison Research on Text Pre-processing Methods on Twitter Sentiment Analysis10.1109/ACCESS.2017.2672677 ↗
31. Reprint of: Computational Approaches for Mining User’S Opinions on the Web 2.010.1016/j.ipm.2014.07.011 ↗
32. Armano, G., Bozzon, A., Cristani, M., and Giuliani, A. (2016, January 8–10). A Comparison between Preprocessing Techniques for Sentiment Analysis in Twitter. Proceedings of the 2nd International Workshop on Knowledge Discovery on the WEB (KDWeb), Cagliari, Italy.
33. A Case-Study for Sentiment Analysis on Twitter
34. Połtyn, M. (2022, July 29). VADER Sentiment Analysis. Available online: https://pypi.org/project/vader-sentiment/.
35. Loria, S. (2022, July 29). TextBlob Documentation. Release 0.15. Available online: https://textblob.readthedocs.io/en/dev/index.html.
36. Aker, A., Petrak, J., and Sabbah, F. (2017, January 2–8). An Extensible Multilingual Open Source Lemmatizer. Proceedings of the International Conference Recent Advances in Natural Language Processing, Varna, Bulgaria.10.26615/978-954-452-049-6_006 ↗
37. Automatic Sarcasm Detection: A Survey10.1145/3124420 ↗
38. Pontiki, M., Galanis, D., Papageorgiou, H., Manandhar, S., and Androutsopoulos, I. (2015, January 4–5). Semeval-2015 Task 12: Aspect Based Sentiment Analysis. Proceedings of the International Workshop on Semantic Evaluation (SemEval 2015), Denver, CO, USA.10.18653/v1/S15-2082 ↗
39. Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., and Stoyanov, V. (2016, January 16–17). Semeval-2016 Task 4: Sentiment Analysis in Twitter. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), San Diego, CA, USA.10.18653/v1/S16-1001 ↗
40. Van Rijsbergen, C. (1979, January 4–7). Information Retrieval: Theory and Practice. Proceedings of the Joint IBM/University of Newcastle upon Tyne Seminar on Data Base Systems, Newcastle upon Tyne, UK.
41. Wikimedia Foundation, Inc (2022, July 29). List of Emoticons. Available online: https://en.wikipedia.org/wiki/List_of_emoticons.
42. (2022, July 29). Internet Slang Dict. Acronym List. Available online: http://www.noslang.com/dictionary/.
43. (2022, July 29). Python Software Foundation. PyEnchant. Available online: https://pypi.org/project/pyenchant/.
44. A Comparison of Text Classification Methods using Different Stemming Techniques10.1504/IJCAT.2019.101171 ↗
45. Appen Ltd (2022, July 29). Datasets Resource Center. Available online: https://appen.com/open-source-datasets/.
46. Flint, J. (2022, July 29). Republican Debate Audience Was the Biggest Ever for a Nonsports Cable Event. The Wall Street Journal. Available online: https://www.wsj.com/articles/republican-debate-audience-was-the-biggest-ever-for-a-nonsports-cable-event-1438992539.
47. Kaggle (2022, July 29). First GOP Debate Twitter Sentiment. Available online: https://www.kaggle.com/crowdflower/first-gop-debate-twitter-sentiment.
48. Concept suggestion engine for professional multimedia archives
49. Mueller, A. (2022, July 29). WordCloud for Python Documentation. Available online: https://amueller.github.io/word_cloud/.
50. Tait, A. (2022, July 29). Why Are Online Jokes Funnier without Punctuation and Capital Letters? The New Statesman. Available online: https://www.newstatesman.com/science-tech/2016/10/why-are-online-jokes-funnier-without-punctuation-and-capital.
51. An Improved Semi-Supervised Dimensionality Reduction Using Feature Weighting: Application to Sentiment Analysis10.1016/j.eswa.2018.05.023 ↗
52. A Comparative Evaluation of Pre-Processing Techniques and their Interactions for Twitter Sentiment Analysis10.1016/j.eswa.2018.06.022 ↗
53. Manning, C.D., Raghavan, P., and Schütze, H. (2008). Introduction to Information Retrieval, Cambridge University Press.10.1017/CBO9780511809071 ↗
54. Sinka, M.P., and Corne, D. (2003). Evolving Better Stoplists for Document Clustering and Web Intelligence. Design and Application of Hybrid Intelligent Systems, IOS Press.
55. Extracting Domain-Specific Stopwords for Text Classifiers10.3233/IDA-150390 ↗
56. Bird, S., Klein, E., and Loper, E. (2009). Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit, O’Reilly Media, Inc.
57. Taggart, C. (2015). New Words for Old: Recycling our Language for the Modern World, Michael O’Mara Books.
58. Wang, H., and Castanon, J.A. (November, January 29). Sentiment Expression via Emoticons on Social Media. Proceedings of the 2015 IEEE International Conference on Big Data (Big Data), Santa Clara, CA, USA.10.1109/BigData.2015.7364034 ↗
59. Creating Emoji Lexica from Unsupervised Sentiment Analysis of Their Descriptions10.1016/j.eswa.2018.02.043 ↗
60. Shah, N., and Rohilla, S. (2022). emot: Open Source Emoticons and Emoji Detection Library, GitHub, Inc.. Available online: https://github.com/NeelShah18/emot.
61. Satapathy, R., Guerreiro, C., Chaturvedi, I., and Cambria, E. (2017, January 18–21). Phonetic-based Microtext Normalization for Twitter Sentiment Analysis. Proceedings of the IEEE International Conference On Data Mining Workshops, New Orleans, LA, USA.10.1109/ICDMW.2017.59 ↗
62. Palomino, M., Grad, D., and Bedwell, J. (2021, January 5–6). GoldenWind at SemEval-2021 Task 5: Orthrus-An Ensemble Approach to Identify Toxicity. Proceedings of the International Workshop on Semantic Evaluation (SemEval-2021), Bangkok, Thailand.10.18653/v1/2021.semeval-1.115 ↗
63. Verma, R. (2022). SMS Slang Translator, GitHub, Inc.
64. Mullen, T., and Malouf, R. (2006, January 27–29). A Preliminary Investigation into Sentiment Analysis of Informal Political Discourse. Proceedings of the AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs, Menlo Park, CA, USA.
65. Carley, K.M., Columbus, D., and Landwehr, P. (2013). AutoMap User’s Guide 2013, Technical Report CMU-ISR-13-105; Carnegie Mellon University, School of Computer Science, Institute for Software Research.10.21236/ADA586976 ↗
66. Barrus, T. (2022, July 29). Pure Python Spell Checker Based on Work by Peter Norvig. Available online: https://pypi.org/project/pyspellchecker/.
67. A normalized Levenshtein distance metric10.1109/TPAMI.2007.1078 ↗
68. The Effect of Stemming and Removal of Stopwords on the Accuracy of Sentiment Analysis on Indonesian-Language Texts
69. A Review: Preprocessing Techniques and Data Augmentation for Sentiment Analysis10.1186/s40649-020-00080-x ↗
70. Haynes, C., Palomino, M.A., Stuart, L., Viira, D., Hannon, F., Crossingham, G., and Tantam, K. (2022). Automatic Classification of National Health Service Feedback. Mathematics, 10.10.3390/math10060983 ↗
71. Loria, S. (2022, July 29). TextBlob: Simplified Text Processing. Available online: https://textblob.readthedocs.io/en/dev/.
72. Vetulani, Z., Paroubek, P., and Kubis, M. (2020). Investigating the Lack of Consensus Among Sentiment Analysis Tools. Human Language Technology. Challenges for Computer Science and Linguistics, Springer International Publishing.10.1007/978-3-030-66527-2 ↗
73. Pattern for python
74. Perkins, J. (2014). Python 3 Text Processing with NLTK 3 Cookbook, Packt Publishing Ltd.
75. Hutto, C.J., and Gilbert, E. (2014, January 1–4). VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proceedings of the 8th International AAAI Conference on Weblogs and Social Media, Ann Arbor, MI, USA.10.1609/icwsm.v8i1.14550 ↗
76. Analyzing the Effectiveness of Semi-Supervised Learning Approaches for Opinion Spam Classification10.1016/j.asoc.2020.107023 ↗
77. Zhang, H., and Su, J. (2004). Naive Bayesian Classifiers for Ranking. European Conference on Machine Learning, Springer.10.1007/978-3-540-30115-8_46 ↗
78. An Overview of Overfitting and Its Solutions10.1088/1742-6596/1168/2/022022 ↗
79. Reliable Accuracy Estimates from k-Fold Cross Validation10.1109/TKDE.2019.2912815 ↗