A Neural Topic Modeling Study Integrating SBERT and Data Augmentation
Source abstract
Abstract
Topic models can extract consistent themes from large corpora for research purposes. In recent years, the combination of pretrained language models and neural topic models has gained attention among scholars. However, this approach has some drawbacks: in short texts, the quality of the topics obtained by the models is low and incoherent, which is caused by the reduced word frequency (insufficient word co-occurrence) in short texts compared to long texts. To address these issues, we propose a neural topic model based on SBERT and data augmentation. First, our proposed easy data augmentation (EDA) method with keyword combination helps overcome the sparsity problem in short texts. Then, the attention mechanism is used to focus on keywords related to the topic and reduce the impact of noise words. Next, the SBERT model is trained on a large and diverse dataset, which can generate high-quality semantic information vectors for short texts. Finally, we perform feature fusion on the augmented data that have been weighted by an attention mechanism with the high-quality semantic information obtained. Then, the fused features are input into a neural topic model to obtain high-quality topics. The experimental results on an English public dataset show that our model generates high-quality topics, with the average scores improving by 2.5% for topic coherence and 1.2% for topic diversity compared to the baseline model.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Advanced Text Analysis Techniques · Computational and Text Analysis Methods
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Information Science and Engineering, Xinjiang University, Urumqi 830046, China
Xinjiang Multilingual Information Technology Laboratory, College of Information Science and Engineering, Xinjiang University, Urumqi 830046, China
Funding
Key Projects of Scientific Research Program of Xinjiang Universities Foundation of China
References (48)
Show source-provided references
1. Latent dirichlet allocation
2. Chaudhary, Y., Gupta, P., Saxena, K., Kulkarni, V., Runkler, T., and Schütze, H. (2020). TopicBERT for energy efficient document classification. arXiv.10.18653/v1/2020.findings-emnlp.152 ↗
3. Sentiment word co-occurrence and knowledge pair feature extraction based LDA short text clustering algorithm10.1007/s10844-020-00597-7 ↗
4. Indexing by latent semantic analysis10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 ↗
5. Hofmann, T. (1999, January 15–19). Probabilistic Latent Semantic Indexing. Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA, USA.10.1145/312624.312649 ↗
6. Sharing clusters among related groups: Hierarchical Dirichlet processes
7. Kingma, D.P., and Welling, M. (2013). Auto-encoding variational bayes. arXiv.
8. Das, R., Zaheer, M., and Dyer, C. (2015, January 26–31). Gaussian LDA for Topic Models with Word Embeddings. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-1077 ↗
9. Wei, X., and Croft, W.B. (2006, January 6–11). LDA-Based Document Models for Ad-Hoc Retrieval. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, USA.10.1145/1148170.1148204 ↗
10. Mehrotra, R., Sanner, S., Buntine, W., and Xie, L. (August, January 28). Improving Lda Topic Models for Microblogs via Tweet Pooling and Automatic Labeling. Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland.
11. Distributed representations of words and phrases and their compositionality
12. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
13. Topic modeling in embedding spaces10.1162/tacl_a_00325 ↗
14. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
15. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.
16. He, P., Liu, X., Gao, J., and Chen, W. (2020). Deberta: Decoding-enhanced bert with disentangled attention. arXiv.
17. Bianchi, F., Terragni, S., Hovy, D., Nozza, D., and Fersini, E. (2020). Cross-lingual contextualized topic models with zero-shot learning. arXiv.10.18653/v1/2021.eacl-main.143 ↗
18. Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv.10.18653/v1/D19-1410 ↗
19. Reimers, N., and Gurevych, I. (2020). Making monolingual sentence embeddings multilingual using knowledge distillation. arXiv.10.18653/v1/2020.emnlp-main.365 ↗
20. Wei, J., and Zou, K. (2019). Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv.10.18653/v1/D19-1670 ↗
21. Mihalcea, R., and Tarau, P. (2004, January 25–26). Textrank: Bringing Order into Text. Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, Barcelona, Spain.
22. Srivastava, A., and Sutton, C. (2017). Autoencoding variational inference for topic models. arXiv.
23. Miao, Y., Yu, L., and Blunsom, P. (2016, January 20–22). Neural Variational Inference for Text Processing. Proceedings of the International Conference on Machine Learning, PMLR, New York, NY, USA.
24. Miao, Y., Grefenstette, E., and Blunsom, P. (2017, January 6–11). Discovering Discrete Latent Topics with Neural Variational Inference. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
25. Card, D., Tan, C., and Smith, N.A. (2017). Neural models for documents with metadata. arXiv.10.18653/v1/P18-1189 ↗
26. Nan, F., Ding, R., Nallapati, R., and Xiang, B. (2019). Topic modeling with wasserstein autoencoders. arXiv.10.18653/v1/P19-1640 ↗
27. Wang, R., Hu, X., Zhou, D., He, Y., Xiong, Y., Ye, C., and Xu, H. (2020). Neural topic modeling with bidirectional adversarial training. arXiv.10.18653/v1/2020.acl-main.32 ↗
28. Wu, J., Rao, Y., Zhang, Z., Xie, H., Li, Q., Wang, F.L., and Chen, Z. (2020, January 5–10). Neural Mixed Counting Models for Dispersed Topic Discovery. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.548 ↗
29. Tian, R., Mao, Y., and Zhang, R. (2020, January 16–20). Learning VAE-LDA Models with Rounded Reparameterization Trick. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.10.18653/v1/2020.emnlp-main.101 ↗
30. Gupta, P., Chaudhary, Y., Buettner, F., and Schütze, H. (February, January 27). Document Informed Neural Autoregressive Topic Models with Distributional Prior. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA.
31. A novel label-based multimodal topic model for social media analysis10.1016/j.dss.2022.113863 ↗
32. Knowledge based topic retrieval for recommendations and tourism promotions
33. Daud, S., Ullah, M., Rehman, A., Saba, T., Damaševičius, R., and Sattar, A. (2023). Topic Classification of Online News Articles Using Optimized Machine Learning Models. Computers, 12.10.3390/computers12010016 ↗
34. Alawadh, H.M., Alabrah, A., Meraj, T., and Rauf, H.T. (2023). English Language Learning via YouTube: An NLP-Based Analysis of Users’ Comments. Computers, 12.10.3390/computers12020024 ↗
35. Awantina, R., and Wibowo, W. (2022, January 1–2). Computational Linguistics Using Latent Dirichlet Allocation for Topic Modeling on Wattpad Review. Proceedings of the 4th International Conference on Science and Technology Applications, ICoSTA 2022, Medan, North Sumatera Province, Indonesia.
36. Liu, Y., and Chen, M. (IEEE Trans. Eng. Manag., 2023). The Knowledge Structure and Development Trend in Artificial Intelligence Based on Latent Feature Topic Model, IEEE Trans. Eng. Manag., early access.10.1109/TEM.2022.3232178 ↗
37. Hoyle, A., Goel, P., and Resnik, P. (2020). Improving neural topic models using knowledge distillation. arXiv.10.18653/v1/2020.emnlp-main.137 ↗
38. Graves, A., and Graves, A. (2012). Supervised Sequence Labelling with Recurrent Neural Networks, Springer.10.1007/978-3-642-24797-2 ↗
39. Dieng, A.B., Wang, C., Gao, J., and Paisley, J. (2016). Topicrnn: A recurrent neural network with long-range semantic dependency. arXiv.
40. Jin, M., Luo, X., Zhu, H., and Zhuo, H.H. (2018, January 1–6). Combining Deep Learning and Topic Modeling for Review Understanding in Context-Aware Recommendation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LO, USA.10.18653/v1/N18-1145 ↗
41. Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., and Brown, D. (2019). Text classification algorithms: A survey. Information, 10.10.3390/info10040150 ↗
42. Aggarwal, C.C., and Zhai, C. (2012). Mining Text Data, Springer.10.1007/978-1-4614-3223-4 ↗
43. Karimi, A., Rossi, L., and Prati, A. (2021). Aeda: An easier data augmentation technique for text classification. arXiv.10.18653/v1/2021.findings-emnlp.234 ↗
44. Terragni, S., Fersini, E., Galuzzi, B.G., Tropeano, P., and Candelieri, A. (2021, January 19–23). Octis: Comparing and Optimizing Topic Models Is Simple!. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Online.10.18653/v1/2021.eacl-demos.31 ↗
45. Lau, J.H., Newman, D., and Baldwin, T. (2014, January 26–30). Machine Reading Tea Leaves: Automatically Evaluating Topic Coherence and Topic Model Quality. Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden.10.3115/v1/E14-1056 ↗
46. Lau, J.H., and Baldwin, T. (2016, January 12–17). The Sensitivity of Topic Coherence Evaluation to Topic Cardinality. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1057 ↗
47. A similarity measure for indefinite rankings10.1145/1852102.1852106 ↗
48. Bianchi, F., Terragni, S., and Hovy, D. (2020). Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. arXiv.10.18653/v1/2021.acl-short.96 ↗