Candidate Term Boundary Conflict Reduction Method for Chinese Geological Text Segmentation
Source abstract
Abstract
Though Chinese word segmentation (CWS) relies heavily on arithmetic power to train huge models and human work to label corpora, models and algorithms are still less accurate, especially for segmentation in a specific domain. In this study, a high-degree-of-freedom-priority candidate term boundary conflict reduction method (HFCR) is proposed to solve the problem of manually setting thresholds on segmentation based on information entropy. We quantify the uncertainty of left and right character connections of candidate terms and then arrange them in descending order for local comparisons to determine term boundaries. Dynamic numerical comparisons are adopted instead of setting a threshold manually and randomly. Experiments show that the average F1-value of CWS for Chinese geological text is higher than 95% and the F1-value for Chinese general datasets is higher than 87%. Compared with representative tokenizers and the SOTA model, our method performs better, which solves the term boundary conflict problem well and has excellent performance on single geological text segmentation without any samples or labels.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Natural Language Processing Techniques · Topic Modeling · Image Processing and 3D Reconstruction
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Key Laboratory of Metallogenic Prediction of Nonferrous Metals and Geological Environment Monitoring, Central South University, Ministry of Education, Changsha 410083, China
School of Geosciences and Info-Physics, Central South University, Changsha 410083, China
Funding
National Natural Science Foundation of China
References (35)
Show source-provided references
1. Discussion on geological science big data and its applications10.1360/N972015-01035 ↗
2. Ma, K. (2018). Research on the Key Technologies of Geological Big Data Representation and Association. [Ph.D. Thesis, China University of Geosciences].
3. Research on segmentation of geological mineral text using conditional random fields
4. Wang, M.G., Li, X.G., Wei, Z., Zhi, S.T., and Wang, H.Y. (2018, January 26–28). Chinese Word Segmentation Based on Deep Learning. Proceedings of the 10th International Conference on Machine Learning and Computing (ICMLC 2018), New York, NY, USA.10.1145/3195106.3195135 ↗
5. Xie, X.J., Xie, Z., Ma, K., Chen, J.G., Qiu, Q.J., Li, H., Pan, S.Y., and Tao, L.F. (2021). Geological Named Entity Recognition based on Bert and BiGRU-Attention-CRF Model. Geol. Bull. China, 1–13. Available online: http://kns.cnki.net/kcms/detail/11.4648.p.20210913.1040.002.html.
6. Tian, Y.H., Song, Y., Xia, F., and Wang, Y.G. (2020, January 5–10). Improving Chinese Word Segmentation with Wordhood Memory Networks. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.10.18653/v1/2020.acl-main.734 ↗
7. An improved Chinese word segmentation algorithm based on dictionary
8. An experimental Study on Dictionary Mechanism for Chinese Word Segmentation
9. Yang, Y. (2005). Mechanical Statistical Word Segmentation System Based on Hash Structure. [Master’s Thesis, Central South University].
10. Improved Chinese word segmentation method based on dictionary
11. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition10.1109/5.18626 ↗
12. Low, J.K., Ng, H.T., and Guo, W.Y. (2005, January 14–15). A maximum entropy approach to Chinese word segmentation. Proceedings of the 4th Sighan Workshop on Chinese Language Processing, Jeju Island, Republic of Korea.
13. McCallum, A., Freitag, D., and Pereira, F. (July, January 29). Maximum Entropy Markov Models for Information Extraction and Segmentation. Proceedings of the 17th International Conference on Machine Learning, Stanford, CA, USA.
14. Zhou, J.S., Dai, X.Y., Ni, R.Y., and Chen, J.J. (2005, January 14–15). A Hybrid Approach to Chinese Word Segmentation around CRFs. Proceedings of the 4th Sighan Workshop on Chinese Language Processing, Jeju Island, Republic of Korea.
15. Natural language processing (almost) from scratch
16. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion
17. Cai, D., and Zhao, H. (2016, January 7–12). Neural Word Segmentation Learning for Chinese. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany.10.18653/v1/P16-1039 ↗
18. Bi-directional Long Short-term Memory Neural Networks for Chinese Word Segmentation
19. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.
20. Zhang, X. (2021). Research on Chinese Word Segmentation for Construction Domain. [Master’s Thesis, Beijing University of Engineering and Architecture].
21. Research on Modeling of Traditional Chinese Medicine Word Segmentation Model Based on Sentencepiece
22. Geological entity recognition method based on Deep Belief Networks
23. A method of Geologic Words Identification Based on Statistics
24. Geological Entity Recognition Based on ELMO-CNN-BILSTM-CRF Model
25. Unsupervised Low-resource Name Entities Recognition in Electric Power Domain
26. Unsupervised keyword extraction model for Chinese single text based on BERT model
27. Domain Adaptive Unsupervised Word Segmentation for Traditional Chinese Medicine Ancient Books
28. Algorithm of n-gram Statistics for Arbitrary n and Knowledge Acquisition Based on Statistics
29. A Chinese Word Segmentation Algorithm Based on Information Entropy
30. An Overview on Information Gain-Based GARC-Type Association Classification
31. Gong, W. (2012). Watershed Model Uncertainty Analysis Based on Information Entropy and Mutual Information. [Ph.D. Thesis, Tsinghua University].
32. Zheng, J.S. (2015). An Image Information Entropy-Based Algorithm of No-Reference Image Quality Assessment. [Master’s Thesis, Beijing Jiaotong University].
33. Link prediction algorithm based on information entropy improved PCA model
34. Study on word Segmentation Method of classical Literature Based on New Word Discovery
35. An Identification Method Text of Overall Commendatory and Derogatory Tendency of Sentences Based on Information Entropy