Benchmarking Deep Learning Methods for Aspect Level Sentiment Classification
Source abstract
Abstract
With the advancements in processing units and easy availability of cloud-based GPU servers, many deep learning-based methods have been proposed for Aspect Level Sentiment Classification (ALSC) literature. With this increase in the number of deep learning methods proposed in ALSC literature, it has become difficult to ascertain the performance difference of one method over the other. To this end, our study provides a statistical comparison of the performance of 35 recent deep learning methods with respect to three performance metrics-Accuracy, Macro F1 score, and Time. The methods are evaluated for eight benchmark datasets. In this study, the statistical comparison is based on Friedman, Nemenyi, and Wilcoxon tests. As per the results of statistical tests, the top-ranking methods could not significantly outperform several other methods in terms of Accuracy and Macro F1 score and performed poorly on-time metric. However, the time taken by any method is crucial to analyze the overall performance. Thus, this study aids the selection of the Deep Learning method, which maximizes the accuracy and Macro F1 score and takes minimal time. Our study also establishes a framework for validating the performance of new and alternate methods in ALSC that can be helpful for researchers and practitioners working in this area.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Text and Document Classification Technologies · Advanced Computing and Algorithms
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
University School of Information, Communication and Technology, Guru Gobind Singh Indraprastha University, Delhi 110078, India
References (57)
Show source-provided references
1. Affective Computing and Sentiment Analysis10.1109/MIS.2016.31 ↗
2. Hu, M., and Liu, B. (2004, January 22–25). Mining and summarizing customer reviews. Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, WA, USA.10.1145/1014052.1014073 ↗
3. Aspect extraction in sentiment analysis: Comparative analysis and survey10.1007/s10462-016-9472-z ↗
4. W2VLDA: Almost unsupervised system for Aspect Based Sentiment Analysis10.1016/j.eswa.2017.08.049 ↗
5. Wagner, J., Arora, P., Cortes, S., Barman, U., Bogdanova, D., Foster, J., and Tounsi, L. (2014, January 23–24). DCU: Aspect-based Polarity Classification for SemEval Task 4. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland.10.3115/v1/S14-2036 ↗
6. Jiang, L., Yu, M., Zhou, M., Liu, X., and Zhao, T. (2011, January 19–24). Target-dependent Twitter Sentiment Classification. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, Portland, OR, USA.
7. Deep Learning for Aspect-Level Sentiment Classification: Survey, Vision and Challenges10.1109/ACCESS.2019.2920075 ↗
8. Deep Learning for Aspect-Based Sentiment Analysis: A Comparative Review10.1016/j.eswa.2018.10.003 ↗
9. Statistical comparisons of classifiers over multiple data sets
10. Nemenyi, P. (1963). Distribution-Free Multiple Comparisons, Princeton University.
11. Statistical Comparison of Modelling Methods for Software Maintainability Prediction10.1142/S0218194013500198 ↗
12. Survey on Aspect-Level Sentiment Analysis10.1109/TKDE.2015.2485209 ↗
13. Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., and Mohammad, A.S. (2016, January 16–17). SemEval-2016 task 5: Aspect based sentiment analysis. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), San Diego, CA, USA.10.18653/v1/S16-1002 ↗
14. Unsupervised Semantic Approach of Aspect-Based Sentiment Analysis for Large-Scale User Reviews10.1109/ACCESS.2020.3042312 ↗
15. Unsupervised word-level affect analysis and propagation in a lexical knowledge graph10.1016/j.knosys.2018.12.017 ↗
16. Rush, A.M., Chopra, S., and Weston, J. (2015, January 17–21). A Neural Attention Model for Abstractive Sentence Summarization. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal.10.18653/v1/D15-1044 ↗
17. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
18. Iyyer, M., Boyd-Graber, J., Claudino, L., Socher, R., and Daum’e, H. (2014, January 25–29). A neural network for factoid question answering over paragraphs. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1070 ↗
19. Cambria, E., Poria, S., Hazarika, D., and Kwok, K. (2018, January 2–7). SenticNet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.11559 ↗
20. Cambria, E., Li, Y., Xing, F.Z., Poria, S., and Kwok, K. (2020, January 19–23). SenticNet 6: Ensemble Application of Symbolic and Subsymbolic AI for Sentiment Analysis. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, Online.10.1145/3340531.3412003 ↗
21. Tang, D., Qin, B., Feng, X., and Liu, T. (2016). Effective LSTMs for Target-Dependent Sentiment Classification. arXiv.
22. Wang, Y., Huang, M., Zhao, L., and Zhu, X. (2016, January 1–5). Attention-based LSTM for Aspect-level Sentiment Classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1058 ↗
23. Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014, January 8–13). Recurrent Models of Visual Attention. Proceedings of the Advances in Neural Information Processing Systems (NIPS), Montreal, QC, Canada.
24. Luong, T., Pham, H., and Manning, C. (2015). Effective Approaches to Attention-based Neural Machine Translation. arXiv.10.18653/v1/D15-1166 ↗
25. Kardakis, S., Perikos, I., Grivokostopoulou, F., and Hatzilygeroudis, I. (2021). Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis. Appl. Sci., 11.10.3390/app11093883 ↗
26. Fan, F., Feng, Y., and Zhao, D. (November, January 31). Multi-grained attention network for aspect-level sentiment classification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.10.18653/v1/D18-1380 ↗
27. Huang, B., Ou, Y., and Carley, K.M. (2018, January 10–13). Aspect Level Sentiment Classification with Attention-over-Attention Neural Networks. Proceedings of the International Conference on Social Computing, Behavioral-Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation, SBP-BRiMS, Washington, DC, USA.10.1007/978-3-319-93372-6_22 ↗
28. Ma, D., Li, S., Zhang, X., and Wang, H. (2017). Interactive Attention Networks for Aspect-Level Sentiment Classification. arXiv.10.24963/ijcai.2017/568 ↗
29. Tang, D., Qin, B., and Liu, T. (2016, January 1–5). Aspect Level Sentiment Classification with Deep Memory Network. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1021 ↗
30. Chen, P., Sun, Z., Bing, L., and Yang, W. (2017, January 7–11). Recurrent Attention Network on Memory for Aspect Sentiment Analysis. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Copenhagen, Denmark.10.18653/v1/D17-1047 ↗
31. Jiang, Q., Chen, L., Xu, R., Ao, X., and Yang, M. (2019, January 3–7). A Challenge Dataset and Effective Models for Aspect-Based Sentiment Analysis. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1654 ↗
32. Enhancing Aspect-Based Sentiment Analysis With Capsule Network10.1109/ACCESS.2020.2997675 ↗
33. Zhang, C., Li, Q., and Song, D. (2019, January 3–7). Aspect-based Sentiment Classification with Aspect-specific Graph Convolutional Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, China.10.18653/v1/D19-1464 ↗
34. Syntactic Edge-Enhanced Graph Convolutional Networks for Aspect-Level Sentiment Classification With Interactive Attention10.1109/ACCESS.2020.3019277 ↗
35. Xu, G., Liu, P., Zhu, Z., Liu, J., and Xu, F. (2021). Attention-Enhanced Graph Convolutional Networks for Aspect-Based Sentiment Classification with Multi-Head Attention. Appl. Sci., 11.10.3390/app11083640 ↗
36. Wang, K., Shen, W., Yang, Y., Quan, X., and Wang, R. (2020, January 5–10). Relational Graph Attention Network for Aspect-based Sentiment Analysis. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.295 ↗
37. Investigating Typed Syntactic Dependencies for Targeted Sentiment Classification Using Graph Attention Neural Network10.1109/TASLP.2020.3042009 ↗
38. Devlin, J., Chang, M.-W., Lee, K., and Toutanov, K. (2019, January 2–7). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT 20194171–4186, Minneapolis, MN, USA.
39. Song, Y., Wang, J., Jiang, T., Liu, Z., and Rao, Y. (2019, January 17–19). Targeted Sentiment Classification with Attentional Encoder Network. Proceedings of the International Conference on Artificial Neural Networks, Munich, Germany.10.1007/978-3-030-30490-4_9 ↗
40. A multi-task learning model for Chinese-oriented aspect polarity classification and aspect term extraction10.1016/j.neucom.2020.08.001 ↗
41. Liu, Q., Zhang, H., Zeng, Y., Huang, Z., and Wu, Z. (2018, January 23–27). Content Attention Model for Aspect Based Sentiment Analysis. Proceedings of the 2018 World Wide Web Conference on World Wide Web, Lyon, France.10.1145/3178876.3186001 ↗
42. Xue, W., and Li, T. (2018, January 15–20). Aspect Based Sentiment Analysis with Gated Convolutional Networks. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1234 ↗
43. Zheng, S., and Xia, R. (2018). Left-center-right separated neural network for aspect-based sentiment analysis with rotatory attention. arXiv.
44. Li, X., Bing, L., Lam, W., and Shi, B. (2018, January 15–20). Transformation networks for target-oriented sentiment classification. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia.10.18653/v1/P18-1087 ↗
45. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
46. Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou, H., Androutsopoulos, I., and Manandhar, S. (2014, January 23–24). SemEval-2014 Task 4: Aspect Based Sentiment Analysis. Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland.10.3115/v1/S14-2004 ↗
47. Pontiki, M., Galanis, D., Papageorgiou, H., Manandhar, S., and Androutsopoulos, I. (2015, January 4–5). Semeval-2015 task 12: Aspect based sentiment analysis. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), Denver, CO, USA.10.18653/v1/S15-2082 ↗
48. Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., and Xu, K. (2014, January 22–24). Adaptive Recursive Neural Networkfor target-dependent twitter sentiment classification. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Baltimore, MD, USA.10.3115/v1/P14-2009 ↗
49. Mitchell, M., Aguilar, J., Wilson, T., and Durme, B.V. (2013, January 18–21). Open Domain Targeted Sentiment. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, WA, USA.
50. Saeidi, M., Bouchard, G., Liakata, M., and Riedel, S. (2016, January 11–16). SentiHood: Targeted Aspect Based Sentiment Analysis Dataset for Urban Neighbourhoods. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, Osaka, Japan.
51. Beyond the Stars: Improving Rating Predictions using Review Text Content
52. Dozat, T., and Manning, C.D. (2017, January 24–26). Deep Biaffine Attention for Neural Dependency Parsing. Proceedings of the ICLR, Toulon, France.
53. Aspect-based sentiment classification with multi-attention network10.1016/j.neucom.2020.01.024 ↗
54. Hollander, M., Wolfe, D.A., and Chicken, E. (2013). Nonparametric Statistical Methods, Wiley. [3rd ed.].
55. Should We Really Use Post-Hoc Tests Based on Mean-Ranks?
56. A critical review of multi-objective optimization in data mining: A position paper10.1145/1046456.1046467 ↗
57. Zhao, X., and Ohsawa, Y. (2018). Sentiment Analysis on the Online Reviews Based on Hidden Markov Model. J. Adv. Inf. Technol., 9.10.12720/jait.9.2.33-38 ↗