Transformer-Based Graph Convolutional Network for Sentiment Analysis
Source abstract
Abstract
Sentiment Analysis is an essential research topic in the field of natural language processing (NLP) and has attracted the attention of many researchers in the last few years. Recently, deep neural network (DNN) models have been used for sentiment analysis tasks, achieving promising results. Although these models can analyze sequences of arbitrary length, utilizing them in the feature extraction layer of a DNN increases the dimensionality of the feature space. More recently, graph neural networks (GNNs) have achieved a promising performance in different NLP tasks. However, previous models cannot be transferred to a large corpus and neglect the heterogeneity of textual graphs. To overcome these difficulties, we propose a new Transformer-based graph convolutional network for heterogeneous graphs called Sentiment Transformer Graph Convolutional Network (ST-GCN). To the best of our knowledge, this is the first study to model the sentiment corpus as a heterogeneous graph and learn document and word embeddings using the proposed sentiment graph transformer neural network. In addition, our model offers an easy mechanism to fuse node positional information for graph datasets using Laplacian eigenvectors. Extensive experiments on four standard datasets show that our model outperforms the existing state-of-the-art models.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Sentiment Analysis and Opinion Mining · Topic Modeling · Advanced Text Analysis Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Computer Science and Engineering, Central South University, Changsha 410083, China
Funding
Fundamental Research Funds for Central Universities of the Central South University · Natural Science Foundation of Hunan Province · National Natural Science Foundation of China
References (69)
Show source-provided references
1. Sentiment analysis using deep learning approaches: An overview10.1007/s11432-018-9941-6 ↗
2. Emotional interpretation using chaotic cuckoo public sentiment variations on textual data from Twitter10.1007/s10772-020-09772-1 ↗
3. Sentiment Analysis Using Multi-Head Attention Capsules With Multi-Channel CNN and Bidirectional GRU10.1109/ACCESS.2021.3073988 ↗
4. Sentiment analysis on movie review through building modified sentiment dictionary by movie genre
5. Sentiment analysis of danmaku videos based on naïve bayes and sentiment dictionary10.1109/ACCESS.2020.2986582 ↗
6. Hasan, A., Moin, S., Karim, A., and Shamshirband, S. (2018). Machine learning-based sentiment analysis for twitter accounts. Math. Comput. Appl., 23.10.3390/mca23010011 ↗
7. What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach10.1016/j.compedu.2019.103724 ↗
8. Jagdale, R.S., Shirsat, V.S., and Deshmukh, S.N. (2019). Sentiment analysis on product reviews using machine learning techniques. Cognitive Informatics and Soft Computing, Springer.10.1007/978-981-13-0617-4_61 ↗
9. Deep learning for sentiment analysis: A survey10.1002/widm.1253 ↗
10. Sentiment analysis using deep learning architectures: A review10.1007/s10462-019-09794-5 ↗
11. Dang, N.C., Moreno-García, M.N., and De la Prieta, F. (2020). Sentiment analysis based on deep learning: A comparative study. Electronics, 9.10.3390/electronics9030483 ↗
12. Drug Adverse Reaction Discovery Based on Attention Mechanism and Fusion of Emotional Information10.3103/S0146411620050053 ↗
13. Peng, Z., Song, H., Kang, B., Moctard, O., He, M., and Zheng, X. (2019, January 2–4). Automatic textual Knowledge Extraction based on Paragraph Constitutive Relations. Proceedings of the 6th International Conference On Systems And Informatics, ICSAI 2019, Shanghai, China.10.1109/ICSAI48974.2019.9010349 ↗
14. Network text sentiment analysis method combining LDA text representation and GRU-CNN10.1007/s00779-018-1183-9 ↗
15. Sentiment analysis of Chinese Weibo based on multi-channel convolutional neural network
16. Teng, Z., Vo, D.T., and Zhang, Y. (2016, January 1–5). Context-sensitive lexicon features for neural sentiment analysis. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA.10.18653/v1/D16-1169 ↗
17. Graph neural networks: A review of methods and applications10.1016/j.aiopen.2021.01.001 ↗
18. Yao, L., Mao, C., and Luo, Y. (February, January 27). Graph convolutional networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
19. Lee, C.M., Narayanan, S.S., and Pieraccini, R. (2002, January 26–29). Classifying emotions in human-machine spoken dialogs. Proceedings of the 2002 IEEE International Conference on Multimedia and Expo, ICME 2002, Lausanne, Switzerland.
20. Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals10.1155/S1110865704406192 ↗
21. Sharaf Al-deen, H.S., Zeng, Z., Al-sabri, R., and Hekmat, A. (2021). An Improved Model for Analyzing Textual Sentiment Based on a Deep Neural Network Using Multi-Head Attention Mechanism. Appl. Syst. Innov., 4.10.3390/asi4040085 ↗
22. Sentiment Analysis and Opinion Mining
23. Kim, S., and Hovy, E.H. (2004, January 23–27). Determining the Sentiment of Opinions. Proceedings of the COLING 2004, 20th International Conference on Computational Linguistics, Geneva, Switzerland.10.3115/1220355.1220555 ↗
24. Jurafsky, D., and Gaussier, É. (2006). Sentiment Retrieval using Generative Models. Proceedings of the EMNLP 2006, 2006 Conference on Empirical Methods in Natural Language Processing, Sydney, Australia, 22–23 July 2006, ACL.
25. Feature selection for sentiment analysis based on content and syntax models10.1016/j.dss.2012.05.023 ↗
26. Selecting Attributes for Sentiment Classification Using Feature Relation Networks10.1109/TKDE.2010.110 ↗
27. A comprehensive survey on word representation models: From classical to state-of-the-art word representation language models10.1145/3434237 ↗
28. Singh, G. (2020). Decision Tree J48 at SemEval-2020 Task 9: Sentiment Analysis for Code-Mixed Social Media Text (Hinglish). arXiv.
29. Liu, B. (2015). Sentiment Analysis-Mining Opinions, Sentiments, and Emotions, Cambridge University Press.10.1017/CBO9781139084789 ↗
30. Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J.R. (2020). A Graph-based Coarse-to-fine Method for Unsupervised Bilingual Lexicon Induction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, 5–10 July 2020, Association for Computational Linguistics.
31. A domain transferable lexicon set for Twitter sentiment analysis using a supervised machine learning approach10.1016/j.eswa.2018.04.006 ↗
32. Modelling Public Sentiment in Twitter: Using Linguistic Patterns to Enhance Supervised Learning
33. Zhang, Y., and Wallace, B.C. (2015). A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification. arXiv.
34. Kando, N., Sakai, T., Joho, H., Li, H., de Vries, A.P., and White, R.W. (2017). Deep Learning for Extreme Multi-label Text Classification. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, 7–11 August 2017, ACM.
35. ImageNet Classification with Deep Convolutional Neural Networks
36. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., and Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, Grand Hyatt, Seattle, Seattle, WA, USA, 18–21 October 2013, ACL. A Meeting of SIGDAT, a Special Interest Group of the ACL.
37. Knight, K., Nenkova, A., and Rambow, O. (2016). Hierarchical Attention Networks for Document Classification. Proceedings of the NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA, 12–17 June 2016, Association for Computational Linguistics.
38. Zong, C., Xia, F., Li, W., and Navigli, R. (2021). Label-Specific Dual Graph Neural Network for Multi-Label Text Classification. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Virtual Event, 1–6 August 2021, Association for Computational Linguistics. (Volume 1: Long Papers).
39. Multi-level graph neural network for text sentiment analysis10.1016/j.compeleceng.2021.107096 ↗
40. Frog-GNN: Multi-perspective aggregation based graph neural network for few-shot text classification10.1016/j.eswa.2021.114795 ↗
41. Inui, K., Jiang, J., Ng, V., and Wan, X. (2019). Text Level Graph Neural Network for Text Classification. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, 3–7 November 2019, Association for Computational Linguistics.
42. Kuchaiev, O., and Ginsburg, B. (2017, January 24–26). Factorization tricks for LSTM networks. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
43. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q.V., Hinton, G.E., and Dean, J. (2017, January 24–26). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
44. Kim, Y., Denton, C., Hoang, L., and Rush, A.M. (2017, January 24–26). Structured Attention Networks. Proceedings of the 5th International Conference on Learning Representations, ICLR 2017, Toulon, France.
45. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural Machine Translation by Jointly Learning to Align and Translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
46. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
47. The Impact of Weighting Schemes and Stemming Process on Topic Modeling of Arabic Long and Short Texts10.1145/3405843 ↗
48. code2vec: Learning distributed representations of code10.1145/3290353 ↗
49. Multi-task learning model based on multi-scale CNN and LSTM for sentiment classification10.1109/ACCESS.2020.2989428 ↗
50. Technical analysis and sentiment embeddings for market trend prediction10.1016/j.eswa.2019.06.014 ↗
51. Wu, Z., Dai, X.Y., Yin, C., Huang, S., and Chen, J. (2018, January 2–7). Improving review representations with user attention and product attention for sentiment classification. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.10.1609/aaai.v32i1.12054 ↗
52. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
53. Murphy, R.L., Srinivasan, B., Rao, V.A., and Ribeiro, B. (2019, January 9–15). Relational Pooling for Graph Representations. Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, CA, USA.
54. You, J., Ying, R., and Leskovec, J. (2019, January 9–15). Position-aware Graph Neural Networks. Proceedings of the 36th International Conference on Machine Learning, ICML 2019, Long Beach, CA, USA.
55. Dwivedi, V.P., Joshi, C.K., Laurent, T., Bengio, Y., and Bresson, X. (2020). Benchmarking Graph Neural Networks. arXiv.
56. Srinivasan, B., and Ribeiro, B. (2020, January 26–30). On the Equivalence between Positional Node Embeddings and Structural Graph Representations. Proceedings of the 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia.
57. Refined Global Word Embeddings Based on Sentiment Concept for Sentiment Analysis10.1109/ACCESS.2021.3062654 ↗
58. Sentiment analysis of comment texts based on BiLSTM10.1109/ACCESS.2019.2909919 ↗
59. Gu, S., Zhang, L., Hou, Y., and Song, Y. (2018, January 20–26). A position-aware bidirectional attention network for aspect-level sentiment analysis. Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, NM, USA.
60. Amplayo, R.K. (2019). Rethinking attribute representation and injection for sentiment classification. arXiv.10.18653/v1/D19-1562 ↗
61. Amplayo, R.K., Kim, J., Sung, S., and Hwang, S.W. (2018). Cold-start aware user and product attention for sentiment classification. arXiv.10.18653/v1/P18-1236 ↗
62. Ma, D., Li, S., Zhang, X., Wang, H., and Sun, X. (December, January 27). Cascading multiway attentions for document-level sentiment classification. Proceedings of the Eighth International Joint Conference on Natural Language Processing, Taipei, Taiwan. (Volume 1: Long Papers).
63. Sentiment classification using a single-layered BiLSTM model10.1109/ACCESS.2020.2988550 ↗
64. Bidirectional LSTM with self-attention mechanism and multi-channel features for sentiment classification10.1016/j.neucom.2020.01.006 ↗
65. Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Zhao, T. (2019). Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization. arXiv.10.18653/v1/2020.acl-main.197 ↗
66. Lai, S., Xu, L., Liu, K., and Zhao, J. (2015, January 25–30). Recurrent convolutional neural networks for text classification. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9513 ↗
67. Wang, Z., Wu, H., Liu, H., and Cai, Q.H. (2020, January 2). Bert-Pair-Networks for Sentiment Classification. Proceedings of the 2020 International Conference on Machine Learning and Cybernetics (ICMLC), Adelaide, Australia.10.1109/ICMLC51923.2020.9469534 ↗
68. Poursepanj, H., Weissbock, J., and Inkpen, D. (2013). uOttawa: System description for semeval 2013 task 2 sentiment analysis in twitter. Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Atlanta, Georgia, 14–15 June 2013, Association for Computational Linguistics.
69. An improved evidence-based aggregation method for sentiment analysis10.1177/0165551519837187 ↗