RoBERTa-GRU: A Hybrid Deep Learning Model for Enhanced Sentiment Analysis
Source abstract
Abstract
This paper proposes a novel hybrid model for sentiment analysis. The model leverages the strengths of both the Transformer model, represented by the Robustly Optimized BERT Pretraining Approach (RoBERTa), and the Recurrent Neural Network, represented by Gated Recurrent Units (GRU). The RoBERTa model provides the capability to project the texts into a discriminative embedding space through its attention mechanism, while the GRU model captures the long-range dependencies of the embedding and addresses the vanishing gradients problem. To overcome the challenge of imbalanced datasets in sentiment analysis, this paper also proposes the use of data augmentation with word embeddings by over-sampling the minority classes. This enhances the representation capacity of the model, making it more robust and accurate in handling the sentiment classification task. The proposed RoBERTa-GRU model was evaluated on three widely used sentiment analysis datasets: IMDb, Sentiment140, and Twitter US Airline Sentiment. The results show that the model achieved an accuracy of 94.63% on IMDb, 89.59% on Sentiment140, and 91.52% on Twitter US Airline Sentiment. These results demonstrate the effectiveness of the proposed RoBERTa-GRU hybrid model in sentiment analysis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Topic Modeling · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Information Science and Technology, Multimedia University, Melaka 75450, Malaysia
Funding
Fundamental Research Grant Scheme of the Ministry of Higher Education · Multimedia University Internal Research Grant
References (26)
Show source-provided references
1. A comprehensive survey on sentiment analysis: Approaches, challenges and trends10.1016/j.knosys.2021.107134 ↗
2. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
3. Advanced classification method of twitter data using sentiment analysis for airline service
4. Makhmudah, U., Bukhori, S., Putra, J.A., and Yudha, B.A.B. (2019, January 16–17). Sentiment Analysis Of Indonesian Homosexual Tweets Using Support Vector Machine Method. Proceedings of the 2019 International Conference on Computer Science, Information Technology, and Electrical Engineering (ICOMITEE), Jember, Indonesia.10.1109/ICOMITEE.2019.8920940 ↗
5. AlSalman, H. (2020, January 19–21). An improved approach for sentiment analysis of arabic tweets in twitter social media. Proceedings of the 2020 3rd International Conference on Computer Applications & Information Security (ICCAIS), Riyadh, Saudi Arabia.10.1109/ICCAIS48893.2020.9096850 ↗
6. Tariyal, A., Goyal, S., and Tantububay, N. (2018, January 28–29). Sentiment Analysis of Tweets Using Various Machine Learning Techniques. Proceedings of the 2018 International Conference on Advanced Computation and Telecommunication (ICACAT), Bhopal, India.10.1109/ICACAT.2018.8933612 ↗
7. Gupta, A., Singh, A., Pandita, I., and Parashar, H. (2019, January 13–15). Sentiment analysis of Twitter posts using machine learning algorithms. Proceedings of the 2019 6th International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India.
8. Jemai, F., Hayouni, M., and Baccar, S. (July, January 28). Sentiment Analysis Using Machine Learning Algorithms. Proceedings of the 2021 International Wireless Communications and Mobile Computing (IWCMC), Harbin, China.10.1109/IWCMC51323.2021.9498965 ↗
9. Ramadhani, A.M., and Goo, H.S. (2017, January 1–2). Twitter sentiment analysis using deep learning methods. Proceedings of the 2017 7th International Annual Engineering Seminar (InAES), Yogyakarta, Indonesia.10.1109/INAES.2017.8068556 ↗
10. Demirci, G.M., Keskin, Ş.R., and Doğan, G. (2019, January 9–12). Sentiment analysis in Turkish with deep learning. Proceedings of the 2019 IEEE International Conference on Big Data, Los Angeles, CA, USA.10.1109/BigData47090.2019.9006066 ↗
11. Raza, G.M., Butt, Z.S., Latif, S., and Wahid, A. (2021, January 20–21). Sentiment Analysis on COVID Tweets: An Experimental Analysis on the Impact of Count Vectorizer and TF-IDF on Sentiment Predictions using Deep Learning Models. Proceedings of the 2021 International Conference on Digital Futures and Transformative Technologies (ICoDT2), Islamabad, Pakistan.10.1109/ICoDT252288.2021.9441508 ↗
12. A CNN-BiLSTM model for document-level sentiment analysis10.3390/make1030048 ↗
13. Tyagi, V., Kumar, A., and Das, S. (2020, January 18–19). Sentiment Analysis on Twitter Data Using Deep Learning approach. Proceedings of the 2020 2nd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), Greater Noida, India.10.1109/ICACCCN51052.2020.9362853 ↗
14. Jang, B., Kim, M., Harerimana, G., Kang, S.u., and Kim, J.W. (2020). Bi-LSTM model to increase accuracy in text classification: Combining Word2vec CNN and attention mechanism. Appl. Sci., 10.10.3390/app10175841 ↗
15. Hossain, N., Bhuiyan, M.R., Tumpa, Z.N., and Hossain, S.A. (2020, January 1–3). Sentiment analysis of restaurant reviews using combined CNN-LSTM. Proceedings of the 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Kharagpur, India.10.1109/ICCCNT49239.2020.9225328 ↗
16. Yang, Y. (2018). Convolutional neural networks with recurrent neural filters. arXiv.10.18653/v1/D18-1109 ↗
17. Harjule, P., Gurjar, A., Seth, H., and Thakur, P. (2020, January 7–8). Text classification on Twitter data. Proceedings of the 2020 3rd International Conference on Emerging Technologies in Computer Engineering: Machine Learning and Internet of Things (ICETCE), Jaipur, India.10.1109/ICETCE48199.2020.9091774 ↗
18. Character-level convolutional networks for text classification
19. Kafle, K., Yousefhussien, M., and Kanan, C. (2017, January 4–7). Data augmentation for visual question answering. Proceedings of the 10th International Conference on Natural Language Generation, Santiago de Compostela, Spain.10.18653/v1/W17-3529 ↗
20. Wang, W.Y., and Yang, D. (2015, January 17–21). That is so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal.10.18653/v1/D15-1306 ↗
21. Maas, A., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., and Potts, C. (2011, January 19–24). Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA.
22. Twitter sentiment classification using distant supervision
23. Dholpuria, T., Rana, Y., and Agrawal, C. (2018, January 24–26). A Sentiment analysis approach through deep learning for a movie review. Proceedings of the 2018 8th International Conference on Communication Systems and Network Technologies (CSNT), Bhopal, India.10.1109/CSNT.2018.8820260 ↗
24. An exploration of airline sentimental tweets with different classification model
25. Hossen, M.S., Jony, A.H., Tabassum, T., Islam, M.T., Rahman, M.M., and Khatun, T. (2021, January 25–27). Hotel review analysis for the prediction of business using deep learning approach. Proceedings of the 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS), Coimbatore, India.10.1109/ICAIS50930.2021.9395757 ↗
26. Vimali, J., and Murugan, S. (2021, January 8–10). A Text Based Sentiment Analysis Model using Bi-directional LSTM Networks. Proceedings of the 2021 6th International Conference on Communication and Electronics Systems (ICCES), Coimbatre, India.10.1109/ICCES51350.2021.9489129 ↗