Examining Attention Mechanisms in Deep Learning Models for Sentiment Analysis
Source abstract
Abstract
Attention-based methods for deep neural networks constitute a technique that has attracted increased interest in recent years. Attention mechanisms can focus on important parts of a sequence and, as a result, enhance the performance of neural networks in a variety of tasks, including sentiment analysis, emotion recognition, machine translation and speech recognition. In this work, we study attention-based models built on recurrent neural networks (RNNs) and examine their performance in various contexts of sentiment analysis. Self-attention, global-attention and hierarchical-attention methods are examined under various deep neural models, training methods and hyperparameters. Even though attention mechanisms are a powerful recent concept in the field of deep learning, their exact effectiveness in sentiment analysis is yet to be thoroughly assessed. A comparative analysis is performed in a text sentiment classification task where baseline models are compared with and without the use of attention for every experiment. The experimental study additionally examines the proposed models’ ability in recognizing opinions and emotions in movie reviews. The results indicate that attention-based models lead to great improvements in the performance of deep neural models showcasing up to a 3.5% improvement in their accuracy.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Topic Modeling · Stock Market Forecasting Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Computer Engineering and Informatics Department, University of Patras, 26504 Patras, Greece
Computer Technology Institute and Press “Diophantus”, 26504 Patras, Greece
References (51)
Show source-provided references
1. Socher, R., Lin, C.C., Manning, C., and Ng, A.Y. (July, January 28). Parsing natural scenes and natural language with recursive neural networks. Proceedings of the 28th International Conference on Machine Learning (ICML-11), Bellevue, WA, USA.
2. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
3. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint.10.3115/v1/D14-1179 ↗
4. Kamath, U., Liu, J., and Whitaker, J. (2019). Deep Learning for NLP and Speech Recognition, Springer.10.1007/978-3-030-14596-5 ↗
5. A CNN-assisted enhanced audio signal processing for speech emotion recognition
6. Clustering-based speech emotion recognition by incorporating learned features and deep BiLSTM10.1109/ACCESS.2020.2990405 ↗
7. Paszkiel, S. (2020). Using neural networks for classification of the changes in the EEG signal based on facial expressions. Analysis and Classification of EEG Signals for Brain–Computer Interfaces, Springer.10.1007/978-3-030-30581-9_7 ↗
8. Aladem, M., and Rawashdeh, S.A. (2020). A single-stream segmentation and depth prediction CNN for autonomous driving. IEEE Intell. Syst.10.1109/MIS.2020.2993266 ↗
9. Voulodimos, A., Doulamis, N., Doulamis, A., and Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Comput. Intell. Neurosci., 2018.10.1155/2018/7068349 ↗
10. Giannopoulos, P., Perikos, I., and Hatzilygeroudis, I. (2018). Deep learning approaches for facial emotion recognition: A case study on FER-2013. Advances in Hybridization of Intelligent Methods, Springer.10.1007/978-3-319-66790-4_1 ↗
11. Att-Net: Enhanced emotion recognition system using lightweight self-attention module10.1016/j.asoc.2021.107101 ↗
12. Emotion recognition using deep learning approach from audio–visual emotional big data10.1016/j.inffus.2018.09.008 ↗
13. Anvarjon, T., and Kwon, S. (2020). Deep-net: A lightweight CNN-based speech emotion recognition system using deep frequency features. Sensors, 20.10.3390/s20185212 ↗
14. Li, Y., and Shen, L. (2018). Skin lesion analysis towards melanoma detection using deep learning network. Sensors, 18.10.3390/s18020556 ↗
15. Liu, J., Chang, W.C., Wu, Y., and Yang, Y. (2017, January 7–11). Deep learning for extreme multi-label text classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Tokyo, Japan.10.1145/3077136.3080834 ↗
16. Sentiment analysis using deep learning architectures: A review10.1007/s10462-019-09794-5 ↗
17. BiLSTM with multi-polarity orthogonal attention for implicit sentiment analysis10.1016/j.neucom.2019.11.054 ↗
18. Dang, N.C., Moreno-García, M.N., and De la Prieta, F. (2020). Sentiment analysis based on deep learning: A comparative study. Electronics, 9.10.3390/electronics9030483 ↗
19. Learning deep architectures for AI10.1561/2200000006 ↗
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, Available online: https://arxiv.org/pdf/1706.03762.pdf.
21. ABCDM: An attention-based bidirectional CNN-RNN deep model for sentiment analysis10.1016/j.future.2020.08.005 ↗
22. Cheng, J., Dong, L., and Lapata, M. (2016). Long short-term memory-networks for machine reading. arXiv.10.18653/v1/D16-1053 ↗
23. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint.
24. Wang, Y., Huang, M., and Zhao, L. (2016, January 1–5). Attention-based lstm for aspect-level sentiment classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1058 ↗
25. Liu, B. (2015). Sentiment Analysis: Mining Opinions, Sentiments, and Emotions, Cambridge University Press. Available online: https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.359.6341&rep=rep1&type=pdf.10.1017/CBO9781139084789 ↗
26. Perikos, I., Kardakis, S., Paraskevas, M., and Hatzilygeroudis, I. (2019, January 29–31). Hidden Markov Models for Sentiment Analysis in Social Media. Proceedings of the 2019 IEEE International Conference on Big Data, Cloud Computing, Data Science & Engineering (BCD), Honolulu, HI, USA.10.1109/BCD.2019.8885272 ↗
27. Recurrent neural network with pooling operation and attention mechanism for sentiment analysis: A multi-task learning approach10.1016/j.knosys.2020.105856 ↗
28. Munkhdalai, T., and Yu, H. (August, January 30). Neural tree indexers for text understanding. Proceedings of the Association for Computational Linguistics, Vancouver, Canada.
29. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016, January 12–17). Hierarchical attention networks for document classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1174 ↗
30. Yin, Y., Song, Y., and Zhang, M. (2017, January 7–11). Document-level multi-aspect sentiment classification as machine comprehension. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1217 ↗
31. Sentiment classification with word localization based on weakly supervised learning with a convolutional neural network10.1016/j.knosys.2018.04.006 ↗
32. Lin, Z., Feng, M., Santos CN, D., Yu, M., Xiang, B., Zhou, B., and Bengio, Y. (2017). A structured self-attentive sentence embedding. arXiv preprint.
33. Chen, H., Sun, M., Tu, C., Lin, Y., and Liu, Z. (2016, January 21). Neural sentiment classification with user and product attention. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1171 ↗
34. Bidirectional LSTM with attention mechanism and convolutional layer for text classification10.1016/j.neucom.2019.01.078 ↗
35. Lexicon-enhanced LSTM with attention for general sentiment analysis10.1109/ACCESS.2018.2878425 ↗
36. Dou, Z.Y. (2017, January 3). Capturing user and product Information for document level sentiment analysis with deep memory network. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark.10.18653/v1/D17-1054 ↗
37. Li, Z., Zhang, Y., Wei, Y., Wu, Y., and Yang, Q. (2017, January 19–25). End-to-end adversarial memory network for cross-domain sentiment classification. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2017), Melbourne, Australia. Available online: https://www.semanticscholar.org/paper/End-to-End-Adversarial-Memory-Network-for-Sentiment-Li-Zhang/85031a4873fe4ddda4a0841b9169b2f164980f3d?p2df.
38. Tang, D., Qin, B., and Liu, T. (2016). Aspect level sentiment classification with deep memory network. arXiv preprint.10.18653/v1/D16-1021 ↗
39. AELA-DLSTMs: Attention-Enabled and Location-Aware Double LSTMs for aspect-level sentiment classification10.1016/j.neucom.2018.11.084 ↗
40. Chen, J., Hu, Y., Liu, J., Xiao, Y., and Jiang, H. (February, January 27). Deep short text classification with knowledge powered attention. Proceedings of the AAAI Conference on Artificial Intelligence 2019, Honolulu, Hawaii.10.1609/aaai.v33i01.33016252 ↗
41. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. arXiv.
42. Dropout: A simple way to prevent neural networks from overfitting
43. Adaptive subgradient methods for online learning and stochastic optimization
44. The vanishing gradient problem during learning recurrent neural nets and problem solutions10.1142/S0218488598000094 ↗
45. Sutskever, I., Vinyals, O., and Le, Q.V. (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, Available online: https://arxiv.org/pdf/1409.3215.pdf.
46. Luong, M.T., Pham, H., and Manning, C.D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint.10.18653/v1/D15-1166 ↗
47. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A., and Potts, C. (2013, January 18–21). Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, DC, USA.
48. Pang, B., and Lee, L. (2005, January 25–30). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, Ann Arbor, MI, USA.10.3115/1219840.1219855 ↗
49. Pang, B., and Lee, L. (2004, January 21–26). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the 42nd annual meeting on Association for Computational Linguistics, Barcelona, Spain.10.3115/1218955.1218990 ↗
50. Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., and Potts, C. (2011, January 19–24). Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, Portland, OR, USA.
51. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Kudlur, M. (2016, January 2–4). Tensorflow: A system for large-scale machine learning. Proceedings of the 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), Savannah, GA, USA.