An ELECTRA-Based Model for Power Safety Named Entity Recognition
Source abstract
Abstract
Power safety named entity recognition (NER) is essential for determining the cause of faults, assessing potential risks, and planning maintenance schedules, contributing to the comprehension and analysis of power safety documentation content and structure. Such analysis is crucial for the development of a knowledge graph within the power safety domain and the augmentation of the associated dataset. This paper introduces a power safety NER model using efficiently learning an encoder that classifies token replacements accurately (ELECTRA) model. This model employs root mean square layer normalization (RMSNorm) and the switched gated linear unit (SwiGLU) activation function, which substitutes the conventional layer normalization (LayerNorm) and the Gaussian error linear units (GeLU). This model also integrates bidirectional long short-term memory (BiLSTM) with conditional random fields (CRF) to bolster performance in NER tasks. Experimental results show that the improved ELECTRA model achieved an F1 value of 93% on the constructed power safety NER dataset. It outperforms the BERT-BiLSTM-CRF model, achieving a 3.3% performance improvement.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Topic Modeling · Advanced Text Analysis Techniques · Natural Language Processing Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Energy and Electrical Engineering, Changchun University of Science and Technology, Changchun 130013, China
School of Control and Computer Engineering, North China Electric Power University, Beijing 100096, China
Funding
Inner Mongolia Electric Power Research Institute
References (33)
Show source-provided references
1. Application and research of knowledge graph in electric power field
2. Sharma, A., Chakraborty, S., and Kumar, S. (2022). Named entity recognition in natural language processing: A systematic review. Proceedings of the Second Doctoral Symposium on Computational Intelligence: DoSCI 2021, Springer.10.1007/978-981-16-3346-1_66 ↗
3. A survey of named entity recognition and classification10.1075/li.30.1.03nad ↗
4. A Survey on Deep Learning for Named Entity Recognition10.1109/TKDE.2020.2981314 ↗
5. Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional LSTM-CRF Models for Sequence Tagging. arXiv.
6. An Introduction to Conditional Random Fields10.1561/2200000013 ↗
7. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. (2016). Neural Architectures for Named Entity Recognition. arXiv.10.18653/v1/N16-1030 ↗
8. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv.
9. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv.
10. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2020). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv.
11. Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., and Wu, H. (2019). ERNIE: Enhanced Representation through Knowledge Integration. arXiv.
12. Clark, K., Luong, M.T., and Le, Q.V. (2020). ELECTRA: Pre-training text encoders as discriminators rather than generators. arXiv.
13. Pakhale, K. (2023). Comprehensive Overview of Named Entity Recognition: Models, Domain-Specific Applications and Challenges. arXiv.
14. Chen, X., Yan, H., Yan, Q., and Zhang, X. Software Entity Recognition Method Based on BERT Embedding. Proceedings of the Machine Learning for Cyber Security: Third International Conference, ML4CS 2020, Guangzhou, China, 8–10 October 2020.10.1007/978-3-030-62460-6 ↗
15. Hu, W., He, L., Ma, H., Wang, K., and Xiao, J. (2022). KGNER: Improving Chinese Named Entity Recognition by BERT Infused with the Knowledge Graph. Appl. Sci., 12.10.3390/app12157702 ↗
16. Yang, R., Gan, Y., and Zhang, C. (2022). Chinese Named Entity Recognition Based on BERT and Lightweight Feature Extraction Model. Information, 13.10.3390/info13110515 ↗
17. A Chinese Named Entity Recognition Model of Maintenance Records for Power Primary Equipment Based on Progressive Multitype Feature Fusion10.1155/2022/8114217 ↗
18. Chen, Y., Liang, Z., Tan, Z., and Lin, D. (2023). Named Entity Recognition in Power Marketing Domain Based on Whole Word Masking and Dual Feature Extraction. Appl. Sci., 13.10.3390/app13169338 ↗
19. Creating Knowledge Graph of Electric Power Equipment Faults Based on BERT–BiLSTM–CRF Model10.1007/s42835-022-01032-3 ↗
20. Fan, X., Zhang, Q., Jia, Q., Lin, J., and Li, C. (2022, January 23–25). Research on Named Entity Recognition Method Based on Deep Learning in Electric Power Public Opinion Field. Proceedings of the 2022 International Conference on Computer Network, Electronic and Automation (ICCNEA), Xi’an, China. ISSN: 2770-7695.10.1109/ICCNEA57056.2022.00040 ↗
21. Study of agricultural finance policy information extraction based on ELECTRA-BiLSTM-CRF10.2478/amns.2021.2.00307 ↗
22. Fu, Y., and Bu, F. (2021, January 22–24). Research on Named Entity Recognition Based on ELECTRA and Intelligent Face Image Processing. Proceedings of the 2021 IEEE International Conference on Emergency Science and Information Technology (ICESIT), Chongqing, China.10.1109/ICESIT53460.2021.9696907 ↗
23. Feng, J., Wang, H., Peng, L., Wang, Y., Song, H., and Guo, H. (2023, January 6–7). Chinese Named Entity Recognition Within the Electric Power Domain. Proceedings of the Emerging Information Security and Applications: 4th International Conference, EISA 2023, Hangzhou, China. ISSN: 1865-0937.
24. Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention is All you Need. arXiv.
25. Zhang, B., and Sennrich, R. (2019). Root Mean Square Layer Normalization. arXiv.
26. Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv.
27. Ba, J.L., Kiros, J.R., and Hinton, G.E. (2016). Layer Normalization. arXiv.
28. Hendrycks, D., and Gimpel, K. (2023). Gaussian Error Linear Units (GELUs). arXiv.
29. Ramachandran, P., Zoph, B., and Le, Q.V. (2017). Searching for Activation Functions. arXiv.
30. Dauphin, Y.N., Fan, A., Auli, M., and Grangier, D. (2017). Language Modeling with Gated Convolutional Networks. arXiv.
31. Glorot, X., Bordes, A., and Bengio, Y. (2011, January 11–13). Deep Sparse Rectifier Neural Networks. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, JMLR, Lauderdale, FL, USA.
32. Eberts, M., and Ulges, A. (2021). Span-based Joint Entity and Relation Extraction with Transformer Pre-training. arXiv.
33. Joint entity recognition and relation extraction as a multi-head selection problem10.1016/j.eswa.2018.07.032 ↗