Text Classification Model Enhanced by Unlabeled Data for LaTeX Formula
Source abstract
Abstract
Generic language models pretrained on large unspecific domains are currently the foundation of NLP. Labeled data are limited in most model training due to the cost of manual annotation, especially in domains including massive Proper Nouns such as mathematics and biology, where it affects the accuracy and robustness of model prediction. However, directly applying a generic language model on a specific domain does not work well. This paper introduces a BERT-based text classification model enhanced by unlabeled data (UL-BERT) in the LaTeX formula domain. A two-stage Pretraining model based on BERT(TP-BERT) is pretrained by unlabeled data in the LaTeX formula domain. A double-prediction pseudo-labeling (DPP) method is introduced to obtain high confidence pseudo-labels for unlabeled data by self-training. Moreover, a multi-rounds teacher–student model training approach is proposed for UL-BERT model training with few labeled data and more unlabeled data with pseudo-labels. Experiments on the classification of the LaTex formula domain show that the classification accuracies have been significantly improved by UL-BERT where the F1 score has been mostly enhanced by 2.76%, and lower resources are needed in model training. It is concluded that our method may be applicable to other specific domains with enormous unlabeled data and limited labelled data.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Information Science and Engineering, East China University of Science and Technology, 130 Meilong Road, Shanghai 200237, China
References (26)
Show source-provided references
1. Yalniz, I.Z., Jégou, H., Chen, K., Paluri, M., and Mahajan, D. (2019). Billion-scale semi-supervised learning for image classification. arXiv.
2. Xie, Q., Luong, M.T., Hovy, E., and Le, Q.V. (2020, January 13–19). Self-training with noisy student improves imagenet classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01070 ↗
3. Kahn, J., Lee, A., and Hannun, A. (2020, January 4–8). Self-training for end-to-end speech recognition. Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Barcelona, Spain.10.1109/ICASSP40776.2020.9054295 ↗
4. Park, D.S., Zhang, Y., Jia, Y., Han, W., Chiu, C., Li, B., Wu, Y., and Le, Q.V. (2020). Improved noisy student training for automatic speech recognition. arXiv.10.21437/Interspeech.2020-1470 ↗
5. Xu, Q., Likhomanenko, T., Kahn, J., Hannun, A., Synnaeve, G., and Collobert, R. (2020). Iterative pseudo-labeling for speech recognition. arXiv.10.21437/Interspeech.2020-1800 ↗
6. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., and Smith, N.A. (2020). Don’t stop pretraining: Adapt language models to domains and tasks. arXiv.10.18653/v1/2020.acl-main.740 ↗
7. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv.10.18653/v1/N18-1202 ↗
8. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
9. BioBERT: A pre-trained biomedical language representation model for biomedical text mining10.1093/bioinformatics/btz682 ↗
10. Peng, S., Yuan, K., Gao, L., and Tang, Z. (2021). MathBERT: A Pre-Trained Model for Mathematical Formula Understanding. arXiv.
11. Alzubaidi, L., Al-Amidie, M., Al-Asadi, A., Humaidi, A.J., Al-Shamma, O., Fadhel, M.A., Zhang, J., Santamaria, J., and Duan, Y. (2021). Novel Transfer Learning Approach for Medical Imaging with Limited Labeled Data. Cancers, 13.10.3390/cancers13071590 ↗
12. Ouali, Y., Hudelot, C., and Tami, M. (2020). An overview of deep semi-supervised learning. arXiv.
13. Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. (2020). Big Self-Supervised Models are Strong Semi-Supervised Learners. arXiv.
14. Xu, Q., Baevski, A., Likhomanenko, T., Tomasello, P., Conneau, A., Collobert, R., Synnaeve, G., and Auli, M. (2021, January 6–11). Self-training and pre-training are complementary for speech recognition. Proceedings of the ICASSP 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2020), Toronto, ON, Canada.10.1109/ICASSP39728.2021.9414641 ↗
15. Iscen, A., Tolias, G., Avrithis, Y., and Chum, O. (2019, January 16–19). Label propagation for deep semi-supervised learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00521 ↗
16. Arazo, E., Ortego, D., Albert, P., O’Connor, N.E., and McGuinness, K. (2020, January 19–24). Pseudo-labeling and confirmation bias in deep semi-supervised learning. Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK.10.1109/IJCNN48605.2020.9207304 ↗
17. McInnes, L., Healy, J., and Melville, J. (2018). Umap: Uniform manifold approximation and projection for dimension reduction. arXiv.10.21105/joss.00861 ↗
18. Deng, Y., Kanervisto, A., Ling, J., and Rush, A.M. (2017, January 6–11). Image-to-markup generation with coarse-to-fine attention. Proceedings of the International Conference on Machine Learning, PMLR, Sydney, Australia.
19. Long short-term memory10.1162/neco.1997.9.8.1735 ↗
20. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.10.3115/v1/D14-1179 ↗
21. Kim, Y. (2014, January 25–29). Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1181 ↗
22. Pennington, J., Socher, R., and Manning, C.D. (2014, January 25–29). Glove: Global vectors for word representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar.10.3115/v1/D14-1162 ↗
23. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., and Dean, J. (2013, January 5–10). Distributed representations of words and phrases and their compositionality. Proceedings of the Advances in Neural Information Processing Systems, Lake Tahoe, CA, USA.
24. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, Long Beach, CA, USA.
25. Maas, A., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., and Potts, C. (2011, January 19–24). Learning word vectors for sentiment analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, OR, USA.
26. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv.