Tibetan Sentence Boundaries Automatic Disambiguation Based on Bidirectional Encoder Representations from Transformers on Byte Pair Encoding Word Cutting Method
Source abstract
Abstract
Sentence Boundary Disambiguation (SBD) is crucial for building datasets for tasks such as machine translation, syntactic analysis, and semantic analysis. Currently, most automatic sentence segmentation in Tibetan adopts the methods of rule-based and statistical learning, as well as the combination of the two, which have high requirements on the corpus and the linguistic foundation of the researchers and are more costly to annotate manually. In this study, we explore Tibetan SBD using deep learning technology. Initially, we analyze Tibetan characteristics and various subword techniques, selecting Byte Pair Encoding (BPE) and Sentencepiece (SP) for text segmentation and training the Bidirectional Encoder Representations from Transformers (BERT) pre-trained language model. Secondly, we studied the Tibetan SBD based on different BERT pre-trained language models, which mainly learns the ambiguity of the shad (“།”) in different positions in modern Tibetan texts and determines through the model whether the shad (“།”) in the texts has the function of segmenting sentences. Meanwhile, this study introduces four models, BERT-CNN, BERT-RNN, BERT-RCNN, and BERT-DPCNN, based on the BERT model for performance comparison. Finally, to verify the performance of the pre-trained language models on the SBD task, this study conducts SBD experiments on both the publicly available Tibetan pre-trained language model TiBERT and the multilingual pre-trained language model (Multi-BERT). The experimental results show that the F1 score of the BERT (BPE) model trained in this study reaches 95.32% on 465,669 Tibetan sentences, nearly five percentage points higher than BERT (SP) and Multi-BERT. The SBD method based on pre-trained language models in this study lays the foundation for establishing datasets for the later tasks of Tibetan pre-training, summary extraction, and machine translation.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Multimodal Machine Learning Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Computer Science & Engineering, Northwest Normal University, Lanzhou 730070, China
Funding
National Natural Science Foundation of China · Major science and technology projects of Gansu province · Science and Technology Commissioner Special Project of Gansu province · Gansu Provincial Department of Education: Industry Support Program Project · Northwest Normal University Young Teachers Research Ability Enhancement Program Project
References (55)
Show source-provided references
1. Kaur, J., and Singh, J. (2019, January 18–19). Deep Neural Network Based Sentence Boundary Detection and End Marker Suggestion for Social Media Text. Proceedings of the 2019 International Conference on Computing, Communication, and Intelligent Systems (ICCCIS), Greater Noida, India.10.1109/ICCCIS48478.2019.8974495 ↗
2. Enriching speech recognition with automatic detection of sentence boundaries and disfluencies10.1109/TASL.2006.878255 ↗
3. Dependency Parsing of Tibetan Compound Sentence
4. Semantic Block Recognition Method for Tibetan Sentences
5. News text classification method and simulation based on the hybrid deep learning model10.1155/2021/8064579 ↗
6. Deep learning based text classification: A comprehensive review10.1145/3439726 ↗
7. Wan, F., and He, X. (2015, January 30–31). Tibetan Syntactic Parsing based on Syllables. Proceedings of the International Conference on Mechatronics & Industrial Informatics, Zhuhai, China.10.2991/icmii-15.2015.130 ↗
8. Garrett, E.J. (2001). Evidentiality and Assertion in Tibetan, University of California.
9. The contribution of corpus linguistics to lexicography and the future of Tibetan dictionaries
10. Segmenting and POS tagging Classical Tibetan using a memory-based tagger
11. Lobsang, G., Lu, W., Honda, K., Wei, J., and Dang, J. (2016, January 13–15). Tibetan vowel analysis with a multi-modal Mandarin-Tibetan speech corpus. Proceedings of the 2016 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), Jeju, Republic of Korea.10.1109/APSIPA.2016.7820776 ↗
12. Wan, F.C., Yu, H.Z., Wu, X.H., and He, X.Z. (2014, January 12–14). Tibetan Syntactic Parsing for Tibetan-Chinese Machine Translation. Proceedings of the 2014 International Conference on Artificial Intelligence and Industrial Application, (AIIA2014), Hong Kong, China.
13. Peters, M., Neumann, M., Iyyer, M., Gardner, M., and Zettlemoyer, L. (2018, January 1–6). Deep contextualized word representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT2018), New Orleans, LA, USA.10.18653/v1/N18-1202 ↗
14. Improving language understanding by generative pre-training
15. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL2019), Minneapolis, MN, USA.
16. Liang, L., Tian, F., and Sun, B.W. (2018, January 2–3). Current status of Tibetan sentiment analysis and cross language analysis. Proceedings of the 2018 6th International Conference on Machinery, Materials and Computing Technology (ICMMCT 2018), Jinan, China.10.2991/icmmct-18.2018.64 ↗
17. Luong, T., Sutskever, I., Le, Q., Vinyals, O., and Zaremba, W. (2015, January 26–31). Addressing the Rare Word Problem in Neural Machine Translation. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China.10.3115/v1/P15-1002 ↗
18. Jean, S., Cho, K., Memisevic, R., and Bengio, Y. (2015, January 26–31). On using very large target vocabulary for neural machine translation. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing (ACL-IJCNLP), Beijing, China.10.3115/v1/P15-1001 ↗
19. Sennrich, R., Haddow, B., and Birch, A. (2016, January 7–12). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany.10.18653/v1/P16-1162 ↗
20. Al-Rfou, R., Choe, D., Constant, N., Guo, M., and Jones, L. (February, January 27). Character-Level Language Modeling with Deeper Self-Attention. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI2019), Honolulu, HI, USA.10.1609/aaai.v33i01.33013159 ↗
21. Wang, C., Cho, K., and Gu, J. (2020, January 7–12). Neural machine translation with byte-level subwords. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i05.6451 ↗
22. Kudo, T. (2018, January 15–20). Subword regularization: Improving neural network translation models with multiple subword candidates. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL2018), Melbourne, Australia.10.18653/v1/P18-1007 ↗
23. Kudo, T., and Richardson, J. (November, January 31). Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP2018), Brussels, Belgium.10.18653/v1/D18-2012 ↗
24. Kim, Y. (2014, January 25–29). Convolutional neural networks for sentence classification. Proceedings of the 19th Empirical Methods in Natural Language Processing (EMNLP2014), Doha, Qatar.10.3115/v1/D14-1181 ↗
25. Liu, P., Qiu, X., and Huang, X. (2016, January 12–17). Recurrent neural network for text classification with multi-task learning. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, Phoenix, AZ, USA.
26. Lai, S., Xu, L., Liu, K., and Zhao, J. (2015, January 25–30). Recurrent convolutional neural networks for text classification. Proceedings of the AAAI Conference on Artificial Intelligence, Austin, TX, USA.10.1609/aaai.v29i1.9513 ↗
27. Johnson, R., and Zhang, T. (August, January 30). Deep Pyramid Convolutional Neural Networks for Text Categorization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics—ACL2017 (Volume 1: Long Papers), Vancouver, BC, Canada.10.18653/v1/P17-1052 ↗
28. Sun, Y., Liu, S.S., Deng, J.J., and Zhao, X.B. (2022, January 9–12). TiBERT: Tibetan pre-trained language model. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC2022), Prague, Czech Republic.
29. Pires, T., Schlinger, E., and Garrette, D. (August, January 28). How multilingual is multilingual BERT?. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Florence, Italy.
30. Decision tree methods: Applications for classification and prediction
31. A novel hybrid multilayer perceptron neural network with improved grey wolf optimizer10.1007/s00521-022-07775-4 ↗
32. Prediction of schizophrenia from activity data using hidden Markov model parameters10.1007/s00521-022-07845-7 ↗
33. Huang, J., and Zweig, G. (2002, January 16–20). Maximum entropy model for punctuation annotation from speech. Proceedings of the 7th International Conference on Spoken Language Processing, ICSLP2002—INTERSPEECH 2002, Denver, CO, USA.10.21437/ICSLP.2002-307 ↗
34. An introduction to conditional random fields10.1561/2200000013 ↗
35. Adaptive multilingual sentence boundary disambiguation
36. Read, J., Dridan, R., Oepen, S., and Solberg, L.J. (2012, January 8–15). Sentence boundary detection: A long solved problem?. Proceedings of the 24th International Conference on Computational Linguistics, Mumbai, India.
37. Mikheev, A. (May, January 29). Tagging sentence boundaries. Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics, Seattle, WA, USA.
38. Periods capitalized words, etc10.1162/089120102760275992 ↗
39. Unsupervised multilingual sentence boundary detection10.1162/coli.2006.32.4.485 ↗
40. Riley, M.D. (1989, January 21–23). Some applications of tree-based modelling to speech and language. Proceedings of the DARPA Speech and Natural Language Workshop, Association for Computational Linguistics, Philadelphia, PA, USA.10.3115/1075434.1075492 ↗
41. Reynar, J.C., and Ratnaparkhi, A. (April, January 31). A maximum entropy approach to identifying sentence boundaries. Proceedings of the 5th Conference on Applied Natural Language Processing, Washington, DC, USA.
42. Gillick, D. (June, January 31). Sentence Boundary Detection and the Problem with the U.S. Human Language Technologies. Proceedings of the Conference of the North American Chapter of the Association of Computational Linguistics, Boulder, CO, USA.
43. Zhao, W.N., Liu, H.D., Yu, X., Wu, J., and Zhang, P. (2010, January 11–13). The Tibetan Sentence Boundary Identification based on Legal Texts. Proceedings of the National Symposium on Computational Linguistics for Young People (YWCL 2010), Wuhan, China.
44. Research on Automatic Recognition Method of Tibetan Sentence Boundary
45. Research on the Automatic Identification of Tibetan Sentence Boundaries with Maximum Entropy Classifier
46. A Maximum Entropy and Rules Approach to Identifying Tibetan Sentence Boundaries
47. Method of Identification of Tibetan Sentence Boundary
48. Modern Tibetan Auxiliary Ending Sentence Boundary Detection
49. Tibetan Sentence Extraction Method Based on Feature of Function Words and Sentence Ending Words
50. Tibetan Sentence Boundary Recognition Based on Mixed Strategy
51. Wan, M.L.Z. (2016). A Tibetan Sentence Boundary Recognition Method Based on Part of Speech at the End of Sentence, Normal University.
52. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017 (NIPS2017), Long Beach, CA, USA.
53. Dual-axial self-attention network for text classification10.1007/s11432-019-2744-2 ↗
54. Study on the construction of Tibetan sentence segmentation dataset under scarcity language resources
55. Koehn, P. (2005, January 12–16). Europarl: A Parallel Corpus for Statistical Machine Translation. Proceedings of the 10th Machine Translation Summit Proceedings of Conference, Phuket, Thailand.