A Study on Performance Enhancement by Integrating Neural Topic Attention with Transformer-Based Language Model
Source abstract
Abstract
As an extension of the transformer architecture, the BERT model has introduced a new paradigm for natural language processing, achieving impressive results in various downstream tasks. However, high-performance BERT-based models—such as ELECTRA, ALBERT, and RoBERTa—suffer from limitations such as poor continuous learning capability and insufficient understanding of domain-specific documents. To address these issues, we propose the use of an attention mechanism to combine BERT-based models with neural topic models. Unlike traditional stochastic topic modeling, neural topic modeling employs artificial neural networks to learn topic representations. Furthermore, neural topic models can be integrated with other neural models and trained to identify latent variables in documents, thereby enabling BERT-based models to sufficiently comprehend the contexts of specific fields. We conducted experiments on three datasets—Movie Review Dataset (MRD), 20Newsgroups, and YELP—to evaluate our model’s performance. Compared to the vanilla model, the proposed model achieved an accuracy improvement of 1–2% for the ALBERT model in multiclassification tasks across all three datasets, while the ELECTRA model showed an accuracy improvement of less than 1%.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Advanced Text Analysis Techniques · Diverse Approaches in Healthcare and Education Studies · Sentiment Analysis and Opinion Mining
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Applied Statistics, Gachon University, 1342 Seongnam-daero, Sujung-gu, Seongnam 13120, Republic of Korea
Funding
National Research Foundation of Korea (NRF)
References (41)
Show source-provided references
1. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
2. Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv.
3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is all you need. arXiv.
4. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv.
5. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
6. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv.
7. Clark, K., Luong, M.T., Le, Q.V., and Manning, C.D. (2020). Electra: Pre-training text encoders as discriminators rather than generators. arXiv.
8. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optim. Bert pretraining approach. arXiv.
9. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., and Anadkat, S. (2023). GPT-4 technical report. arXiv.
10. He, P., Gao, J., and Chen, W. (2021). DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. arXiv.
11. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., and Azhar, F. (2023). LLaMA: Open and efficient foundation language models. arXiv.
12. Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. arXiv.10.18653/v1/P19-1355 ↗
13. Pre-trained models: Past, present and future10.1016/j.aiopen.2021.08.002 ↗
14. Latent dirichlet allocation
15. Zhao, H., Phung, D., Huynh, V., Jin, Y., Du, L., and Buntine, W. (2021). Topic modelling meets deep neural networks: A survey. arXiv.10.24963/ijcai.2021/638 ↗
16. Supervised topic models
17. Ramage, D., Hall, D., Nallapati, R., and Manning, C.D. (2009, January 6–7). Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, Singapore.10.3115/1699510.1699543 ↗
18. Moghaddam, S., and Ester, M. (2011, January 24–28). ILDA: Interdependent LDA model for learning latent aspects and their ratings from online product reviews. Proceedings of the 34th International ACM SIGIR Conference on RESEARCH and Development in Information Retrieval, Beijing, China.10.1145/2009916.2010006 ↗
19. Supervised topic models for multi-label classification10.1016/j.neucom.2014.07.053 ↗
20. Li, J., Cardie, C., and Li, S. Topicspam: A topic-model based approach for spam detection. Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
21. Multilingual topic modeling for tracking COVID-19 trends based on Facebook data analysis10.1007/s10489-020-02033-3 ↗
22. A hybrid user-based collaborative filtering algorithm with topic model10.1007/s10489-021-02207-7 ↗
23. Gupta, P., Chaudhary, Y., Buettner, F., and Schütze, H. (February, January 27). Document informed neural autoregressive topic models with distributional prior. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA. No. 01.
24. A neural autoregressive topic model
25. Atm: Adversarial-neural topic model10.1016/j.ipm.2019.102098 ↗
26. Wang, R., Hu, X., Zhou, D., He, Y., Xiong, Y., Ye, C., and Xu, H. (2002). Neural topic modeling with bidirectional adversarial training. arXiv.
27. Miao, Y., Grefenstette, E., and Blunsom, P. (2017, January 6–11). Discovering discrete latent topics with neural variational inference. Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia.
28. Wu, J., Rao, Y., Zhang, Z., Xie, H., Li, Q., Wang, F.L., and Chen, Z. Neural mixed counting models for dispersed topic discovery. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
29. Wang, X., and Yang, Y. (2020, January 26–28). Neural topic model with attention for supervised learning. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, Online.
30. Peinelt, N., Nguyen, D., and Liakata, M. tBERT: Topic models and BERT joining forces for semantic similarity detection. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
31. Yin, J., and Wang, J. A dirichlet multinomial mixture model-based approach for short text clustering. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
32. Chaudhary, Y., Gupta, P., Saxena, K., Kulkarni, V., Runkler, T., and Schütze, H. (2020). TopicBERT for energy efficient document classification. arXiv.10.18653/v1/2020.findings-emnlp.152 ↗
33. Miao, Y., Yu, L., and Blunsom, P. (2016, January 19–24). Neural variational inference for text processing. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA.
34. Bianchi, F., Terragni, S., and Hovy, D. (2020). Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. arXiv.10.18653/v1/2021.acl-short.96 ↗
35. Glorot, X., and Bengio, Y. (2010, January 13–15). Understanding the difficulty of training deep feedforward neural networks. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Sardinia, Italy.
36. Pang, B., and Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv.10.3115/1219840.1219855 ↗
37. Kim, Y. (2014). Convolutional neural networks for sentence classification. arXiv.10.3115/v1/D14-1181 ↗
38. (2024, July 23). Home Page for 20Newsgroups Data Set. Available online: http://qwone.com/~jason/20Newsgroups/.
39. Text classification based on deep belief network and softmax regression10.1007/s00521-016-2401-x ↗
40. (2024, July 23). Yelp Open Dataset. Available online: https://www.yelp.com/dataset.
41. Tang, D., Qin, B., and Liu, T. Document modeling with gated recurrent neural network for sentiment classification. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.10.18653/v1/D15-1167 ↗