An Improved Sentiment Classification Approach for Measuring User Satisfaction toward Governmental Services’ Mobile Apps Using Machine Learning Methods with Feature Engineering and SMOTE Technique
Source abstract
Abstract
Analyzing the sentiment of Arabic texts is still a big research challenge due to the special characteristics and complexity of the Arabic language. Few studies have been conducted on Arabic sentiment analysis (ASA) compared to English or other Latin languages. In addition, most of the existing studies on ASA analyzed datasets collected from Twitter. However, little attention was given to the huge amounts of reviews for governmental or commercial mobile applications on Google Play or the App Store. For instance, the government of Saudi Arabia developed several mobile applications in healthcare, education, and other sectors as a response to the COVID-19 pandemic. To address this gap, this paper aims to analyze the users’ opinions of six applications in the healthcare sector. An improved sentiment classification approach was proposed for measuring user satisfaction toward governmental services’ mobile apps using machine learning models with different preprocessing methods. The Arb-AppsReview dataset was collected from the reviews of these six mobile applications available on Google Play and the App Store, which includes 51k reviews. Then, several feature engineering approaches were applied, which include Bing Liu lexicon, AFINN, and MPQA Subjectivity Lexicon, bag of words (BoW), term frequency-inverse document frequency (TF-IDF), and the Google pre-trained Word2Vec. Additionally, the SMOTE technique was applied as a balancing technique on this dataset. Then, five ML models were applied to classify the sentiment opinions. The experimental results showed that the highest accuracy score (94.38%) was obtained by applying a support vector machine (SVM) using the SMOTE technique with all concatenated features.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Spam and Phishing Detection · Hate Speech and Cyberbullying Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Information Technology, College of Computer, Qassim University, Buraydah 51452, Saudi Arabia
Department of Computer Science, College of Applied Sciences, Taiz University, Taiz 6803, Yemen
College of Computer Science and Engineering, Taibah University, Medina 42353, Saudi Arabia
Department of Computer Science, Saba’a Region University, Mareb, Yemen
DAAI Research Group, Department of Computing and Data Science, School of Computing and Digital Technology, Birmingham City University, Birmingham B4 7XG, UK
Department of Computer Science, College of Computer, Qassim University, Buraydah 51452, Saudi Arabia
Funding
Deanship of Scientific Research at the University of Qassim, Saudi Arabia
References (49)
Show source-provided references
1. Ensemble of feature sets and classification algorithms for sentiment classification10.1016/j.ins.2010.11.023 ↗
2. A study on sentiment analysis techniques of Twitter data
3. Alomari, K.M., ElSherif, H.M., and Shaalan, K. (2017). Arabic tweets sentimental analysis using machine learning. International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Springer.10.1007/978-3-319-60042-0_66 ↗
4. Abuelenin, S., Elmougy, S., and Naguib, E. (2017). Twitter sentiment analysis for arabic tweets. International Conference on Advanced Intelligent Systems and Informatics, Springer.10.1007/978-3-319-64861-3_44 ↗
5. Shoukry, A., and Rafea, A. (2012, January 21–25). Sentence-level Arabic sentiment analysis. Proceedings of the 2012 International Conference on Collaboration Technologies and Systems (CTS), Denver, CO, USA.10.1109/CTS.2012.6261103 ↗
6. Wang, Y., Huang, M., Zhu, X., and Zhao, L. (2016, January 1–4). Attention-based LSTM for aspect-level sentiment classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1058 ↗
7. Abdullah, M., Hadzikadicy, M., and Shaikhz, S. (2018, January 17–20). SEDAT: Sentiment and emotion detection in Arabic text using CNN-LSTM deep learning. Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, USA.10.1109/ICMLA.2018.00134 ↗
8. Sentiment analysis in Arabic: A review of the literature10.1016/j.asej.2017.04.007 ↗
9. OCA: Opinion corpus for Arabic10.1002/asi.21598 ↗
10. Arabic Sentiment Analysis: A Systematic Literature Review
11. Tsarfaty, R., Seddah, D., Goldberg, Y., Kübler, S., Versley, Y., Candito, M., and Tounsi, L. (2010, January 5). Statistical parsing of morphologically rich languages (spmrl) what, how and whither. Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, Los Angeles, CA, USA.
12. Elouardighi, A., Maghfour, M., Hammia, H., and Aazi, F.-Z. (2017, January 24–26). A machine Learning approach for sentiment analysis in the standard or dialectal Arabic Facebook comments. Proceedings of the 2017 3rd International Conference of Cloud Computing Technologies and Applications (CloudTech), Rabat, Morocco.10.1109/CloudTech.2017.8284706 ↗
13. An approach for detecting spam in Arabic opinion reviews
14. Data and Text Mining Techniques for Classifying Arabic Tweet Polarity
15. Lexicon-based sentiment analysis of arabic tweets: A survey
16. Text Mining Techniques for Sentiment Analysis of Arabic Dialects: Literature Review10.25046/aj0601112 ↗
17. Alotaibi, S., Mehmood, R., and Katib, I. (2019, January 10–13). Sentiment analysis of arabic tweets in smart cities: A review of saudi dia-lect. Proceedings of the 2019 Fourth International Conference on Fog and Mobile Edge Computing (FMEC), Rome, Italy.10.1109/FMEC.2019.8795331 ↗
18. An enhanced approach for arabic sentiment analysis
19. Implementation of machine learning algorithms in Ara-bic sentiment analysis using N-gram features10.1016/j.procs.2019.06.048 ↗
20. Studying the effect of characteristic vector alteration on Arabic sentiment classification
21. Sentiment analysis for arabic tweets da-tasets: Lexicon-based and machine learning approaches
22. Aljameel, S.S., Alabbad, D.A., Alzahrani, N.A., AlQarni, S.M., AlAmoudi, F.A., Babili, L.M., Aljaafary, S.K., and Alshamrani, F.M. (2020). A Sentiment Analysis Approach to Predict an Individual’s Awareness of the Precautionary Procedures to Prevent COVID-19 Outbreaks in Saudi Arabia. Int. J. Environ. Res. Public Health, 18.10.3390/ijerph18010218 ↗
23. Arabic Tweets Sentiment Analysis about Online Learning during COVID-19 in Saudi Arabia
24. Alassaf, M., and Qamar, A.M. Improving sentiment analysis of Arabic tweets by One-Way ANOVA, J. King Saud Univ. Comput. Inf. Sci., in press.
25. Sentiment Analysis of Arabic Tweets using Deep Learning10.1016/j.procs.2018.10.466 ↗
26. Surface and Deep Features Ensemble for Sentiment Analysis of Arabic Tweets10.1109/ACCESS.2019.2924314 ↗
27. Deep learning approaches for Arabic sentiment analysis10.1007/s13278-019-0596-4 ↗
28. Deep learning for emotion analysis in Arabic tweets10.1186/s40537-021-00523-w ↗
29. Alharbi, N.H., and Alkhateeb, J.H. (2021, January 4–5). Sentiment Analysis of Arabic Tweets Related to COVID-19 Using Deep Neural Network. Proceedings of the 2021 International Congress of Advanced Technology and Engineering (ICOTEN), Virtual Conference.10.1109/ICOTEN52080.2021.9493467 ↗
30. Shahi, A.M., Issac, B., and Modapothala, J.R. (2012, January 6–9). Intelligent Corporate Sustainability report scoring solution using machine learning approach to text categorization. Proceedings of the 2012 IEEE Conference on Sustainable Utilization and Development in Engineering and Technology (STUDENT), Kuala Lumpur, Malaysia.10.1109/STUDENT.2012.6408409 ↗
31. Sentiment analysis of tweets using a uni-fied convolutional neural network-long short-term memory network model10.1111/coin.12415 ↗
32. Al-Sarem, M., Saeed, F., Al-Mekhlafi, Z.G., Mohammed, B.A., Hadwan, M., Al-Hadhrami, T., Alshammari, M.T., Alreshidi, A., and Alshammari, T.S. (2021). An Improved Multiple Features and Machine Learning-Based Approach for Detecting Clickbait News on Social Networks. Appl. Sci., 11.10.3390/app11209487 ↗
33. Al-Sarem, M., Al-Harby, M., Saeed, F., and Hezzam, E.A. Machine Learning Classifiers with Preprocessing Techniques for Rumor Detection on Social Media: An Empirical Study. Int. J. Cloud Computing., 2021. in press.10.1504/IJCC.2022.124797 ↗
34. Ensemble Methods for Instance-Based Arabic Language Authorship Attribution10.1109/ACCESS.2020.2964952 ↗
35. Rustam, F., Ashraf, I., Mehmood, A., Ullah, S., and Choi, G.S. (2019). Tweets Classification on the Base of Sentiments for US Airline Companies. Entropy, 21.10.3390/e21111078 ↗
36. Gaye, B., Zhang, D., and Wulamu, A. (2021). A Tweet Sentiment Classification Approach Using a Hybrid Stacked Ensemble Technique. Information, 12.10.3390/info12090374 ↗
37. Textblob Documentation
38. Hu, M., and Liu, B. (2004, January 22–25). Mining and summarizing customer reviews. Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA.10.1145/1014052.1014073 ↗
39. Nielsen, F.Å. (2011). A new ANEW: Evaluation of a word list for sentiment analysis in microblogs. arXiv.
40. Wilson, T., Wiebe, J., and Hoffmann, P. (2005, January 6–8). Recognizing contextual polarity in phrase-level sentiment analysis. Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Vancouver, BC, Canada.10.3115/1220575.1220619 ↗
41. Heaton, J. (April, January 30). An empirical analysis of feature engineering for predictive modeling. Proceedings of the IEEE SoutheastCon 2016, Norfolk, VA, USA.10.1109/SECON.2016.7506650 ↗
42. Al-Sarem, M., Saeed, F., Boulila, W., Emara, A.H., Al-Mohaimeed, M., and Errais, M. (2020). Feature Selection and Classification Using CatBoost Method for Improving the Performance of Predicting Parkinson’s Disease. Advances on Smart and Soft Computing, Springer.10.1007/978-981-15-6048-4_17 ↗
43. Al-Sarem, M., Alsaeedi, A., Saeed, F., Boulila, W., and AmeerBakhsh, O. (2021). A Novel Hybrid Deep Learning Model for De-tecting COVID-19-Related Rumors on Social Media Based on LSTM and Concatenated Parallel CNNs. Appl. Sci., 11.10.3390/app11177940 ↗
44. Detecting Rumors on Social Media Based on a CNN Deep Learning Technique10.1007/s13369-020-04839-2 ↗
45. Sentiment Analysis of Review Data Using Blockchain and LSTM to Improve Regulation for a Sustainable Market10.3390/jtaer17010001 ↗
46. Abo, M.E.M., Idris, N., Mahmud, R., Qazi, A., Hashem, I.A.T., Maitama, J.Z., and Yang, S. (2021). A Multi-Criteria Ap-proach for Arabic Dialect Sentiment Analysis for Online Reviews: Exploiting Optimal Machine Learning Algorithm Selection. Sustainability, 13.10.3390/su131810018 ↗
47. Arabic Sentiment Analysis of Users’ Opinions of Govern-mental Mobile Applications
48. Impact of SMOTE on Imbalanced Text Features for Toxic Comments Classification Using RVVC Model10.1109/ACCESS.2021.3083638 ↗
49. SMOTE: Synthetic Minority Over-sampling Technique10.1613/jair.953 ↗