Structure-Aware and Format-Enhanced Transformer for Accident Report Modeling
Source abstract
Abstract
Modeling accident investigation reports is crucial for elucidating accident causation mechanisms, analyzing risk evolution processes, and formulating effective accident prevention strategies. However, such reports are typically long, hierarchically structured, and information-dense, posing unique challenges for existing language models. To address these domain-specific characteristics, this study proposes SAFE-Transformer, a Structure-Aware and Format-Enhanced Transformer designed for long-document modeling in the emergency safety context. SAFE-Transformer adopts a dual-stream encoding architecture to separately model symbolic section features and heading text, integrates hierarchical depth and format types into positional encodings, and introduces a dynamic gating unit to adaptively fuse headings with paragraph semantics. We evaluate the model on a multi-label accident intelligence classification task using a real-world corpus of 1632 official reports from high-risk industries. Results demonstrate that SAFE-Transformer effectively captures hierarchical semantic structure and outperforms strong long-text baselines. Further analysis reveals an inverted U-shaped performance trend across varying report lengths and highlights the role of attention sparsity and label distribution in long-text modeling. This work offers a practical solution for structurally complex safety documents and provides methodological insights for downstream applications in safety supervision and risk analysis.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Occupational Health and Safety Research · Topic Modeling · Risk and Safety Analysis
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Electric Power Engineering, South China University of Technology, Guangzhou 510641, China
Shenzhen Urban Public Safety and Technology Institute, Shenzhen 518024, China
Shenzhen Research Institute, China University of Mining and Technology, Shenzhen 518057, China
School of Mechanics and Civil Engineering, China University of Mining and Technology, Xuzhou 221116, China
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China
Funding
Shenzhen Science and Technology Program
References (70)
Show source-provided references
1. An Improved Text Mining Approach to Extract Safety Risk Factors from Construction Accident Reports10.1016/j.ssci.2021.105216 ↗
2. Pandithawatta, S., Ahn, S., Rameezdeen, R., Chow, C.W.K., and Gorjian, N. (2024). Systematic Literature Review on Knowledge-Driven Approaches for Construction Safety Analysis and Accident Prevention. Buildings, 14.10.3390/buildings14113403 ↗
3. Recursive XY Cut Using Bounding Boxes of Connected Components10.1109/ICDAR.1995.602059 ↗
4. A Fast and Efficient Method for Extracting Text Paragraphs and Graphics from Unconstrained Documents
5. The Document Spectrum for Page Layout Analysis10.1109/34.244677 ↗
6. Huang, Y., Lv, T., Cui, L., Lu, Y., and Wei, F. (2022, January 10–14). Layoutlmv3: Pre-training for Document AI With Unified Text and Image Masking. Proceedings of the 30th ACM International Conference on Multimedia, Lisboa, Portugal.10.1145/3503161.3548112 ↗
7. Learning Nongenerative Grammatical Models for Document Analysis10.1109/ICCV.2005.140 ↗
8. Kd-trees for Document Layout Analysis
9. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., and Polosukhin, I. (2017, January 4–9). Attention Is All You Need. Proceedings of the Advances in Neural Information Processing Systems 30, Long Beach, CA, USA.
10. Devlin, J., Chang, M.W., Lee, K., and Toutanova, K. (2019, January 2–7). Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, MN, USA.
11. Convolutional Neural Network Algorithm–Based Novel Automatic Text Classification Framework for Construction Accident Reports10.1061/JCEMD4.COENG-13523 ↗
12. Using Text Mining to Establish Knowledge Graph from Accident/Incident Reports in Risk Assessment10.1016/j.eswa.2022.117991 ↗
13. Automated Text Classification of Near-Misses from Safety Reports: An Improved Deep Learning Approach10.1016/j.aei.2020.101060 ↗
14. Ray, U., Arteaga, C., and Ahn, Y. (2024). Enhanced Identification of Equipment Failures from Descriptive Accident Reports Using Language Generative Model. Eng. Constr. Archit. Manag., ahead of print.10.1108/ECAM-09-2024-1259 ↗
15. Automatic Construction Accident Report Analysis Using Large Language Models (LLMs)10.26599/JIC.2024.9180039 ↗
16. Sun, C., Qiu, X., Xu, Y., and Huang, X. (2019, January 18–20). How to Fine-Tune Bert for Text Classification?. Proceedings of the China National Conference on Chinese Computational Linguistics, Kunming, China.10.1007/978-3-030-32381-3_16 ↗
17. Adhikari, A., Ram, A., Tang, R., and Lin, J. (2019). Docbert: Bert for Document Classification. arXiv.
18. Mosbach, M., Andriushchenko, M., and Klakow, D. (2020). On the Stability of Fine-Tuning Bert: Misconceptions, Explanations, and Strong Baselines. arXiv.
19. Identifying Incident Causal Factors to Improve Aviation Transportation Safety: Proposing a Deep Learning Approach
20. Wang, J., and Yan, M. (2019). Application of an Improved Model for Accident Analysis: A Case Study. Int. J. Environ. Res. Public Health, 16.10.3390/ijerph16152756 ↗
21. Shayboun, M. (2022). Toward Accident Prevention Through Machine Learning Analysis of Accident Reports. [Master’s Thesis, Universidade Tecnica de Lisboa].
22. Analyses of Systems Theory for Construction Accident Prevention with Specific Reference to OSHA Accident Reports10.1016/j.ijproman.2012.12.004 ↗
23. Lombardi, M., Mauro, F., Fargnoli, M., Napoleoni, Q., Berardi, D., and Berardi, S. (2023). Occupational Risk Assessment in Landfills: Research Outcomes from Italy. Safety, 9.10.3390/safety9010003 ↗
24. Identification of Systems Thinking Aspects in ISO 45001: 2018 on Occupational Health & Safety Management10.1016/j.ssci.2022.105671 ↗
25. Marhavilas, P.K., Pliaki, F., and Koulouriotis, D. (2022). International Management System Standards Related to Occupational Safety and Health: An Updated Literature Survey. Sustainability, 14.10.3390/su142013282 ↗
26. Chalkidis, I., Fergadiotis, M., Kotitsas, S., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020). An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels. arXiv.10.18653/v1/2020.emnlp-main.607 ↗
27. Dai, X., Chalkidis, I., Darkner, S., and Elliott, D. (2022). Revisiting Transformer-Based Models for Long Document Classification. arXiv.10.18653/v1/2022.findings-emnlp.534 ↗
28. Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., and Hovy, E. (2016, January 12–17). Hierarchical Attention Networks for Document Classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, CA, USA.10.18653/v1/N16-1174 ↗
29. Wu, C., Wu, F., Qi, T., and Huang, Y. (2021). Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling. arXiv.10.18653/v1/2021.acl-short.107 ↗
30. Liu, Y., Liu, J., Chen, L., Lu, Y., Feng, S., Feng, Z., and Wang, H. (2022). Ernie-Sparse: Learning Hierarchical Efficient Transformer Through Regularized Self-Attention. arXiv.
31. Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., and Ahmed, A. (2020, January 6–12). Big Bird: Transformers for Longer Sequences. Proceedings of the Advances in Neural Information Processing Systems 33, Online.
32. Beltagy, I., Peters, M.E., and Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv.
33. A Document Classification Algorithm Using the Fuzzy Set Theory and Hierarchical Structure of Documents
34. Reason, J. (2016). Managing the Risks of Organizational Accidents, Routledge.10.4324/9781315543543 ↗
35. Leveson, N., Daouk, M., Dulac, N., and Marais, K. (1998). Applying STAMP in Accident Analysis. NASA Conference Publication, NASA.
36. Analysis of Accident Reports10.1080/00140138208925058 ↗
37. Modeling Traffic Accident Occurrence and Involvement10.1016/S0001-4575(99)00094-9 ↗
38. Incomplete Accident Reporting: Meta-Analysis of Studies Made in 13 Countries10.3141/1665-18 ↗
39. Artificial Intelligence and Inequality: Challenges and Opportunities
40. Fiegler-Rudol, J., Lau, K., Mroczek, A., and Kasperczyk, J. (2025). Exploring Human–AI Dynamics in Enhancing Workplace Health and Safety: A Narrative Review. Int. J. Environ. Res. Public Health, 22.10.20944/preprints202501.0483.v1 ↗
41. Construction Accident Narrative Classification: An Evaluation of Text Mining Techniques10.1016/j.aap.2017.08.026 ↗
42. Construction Site Accident Analysis Using Text Mining and Natural Language Processing Techniques10.1016/j.autcon.2018.12.016 ↗
43. Comberti, L., Demichela, M., Baldissone, G., Fois, G., and Luzzi, R. (2018). Large Occupational Accidents Data Analysis with a Coupled Unsupervised Algorithm: The SOM k-Means Method. An Application to the Wood Industry. Safety, 4.10.3390/safety4040051 ↗
44. Lombardi, M., Fargnoli, M., and Parise, G. (2019). Risk Profiling from the European Statistics on Accidents at Work (ESAW) Accidents’ Databases: A Case Study in Construction Sites. Int. J. Environ. Res. Public Health, 16.10.3390/ijerph16234748 ↗
45. Li, J., and Wu, C. (2023). Deep Learning and Text Mining: Classifying and Extracting Key Information from Construction Accident Narratives. Appl. Sci., 13.10.3390/app131910599 ↗
46. Zhang, J., Zi, L., and Hou, Y. (2020). A C-BiLSTM Approach to Classify Construction Accident Reports. Appl. Sci., 10.10.3390/app10175754 ↗
47. Construction-Accident Narrative Classification Using Shallow and Deep Learning10.1061/(ASCE)CO.1943-7862.0002354 ↗
48. Chalkidis, I. (2023). ChatGPT May Pass the Bar Exam Soon, but Has a Long Way to Go for the Lexglue Benchmark. arXiv.10.2139/ssrn.4385460 ↗
49. Tan, B., Yang, Z., AI-Shedivat, M., Xing, E.P., and Hu, Z. (2020). Progressive Generation of Long Text with Pretrained Language Models. arXiv.10.18653/v1/2021.naacl-main.341 ↗
50. Han, G., Tsao, J., and Huang, X. (2024). Length-Aware Multi-Kernel Transformer for Long Document Classification. arXiv.10.18653/v1/2024.starsem-1.22 ↗
51. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., and Zettlemoyer, L. (2019). Bart: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension. arXiv.10.18653/v1/2020.acl-main.703 ↗
52. Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., and Androutsopoulos, I. (2020). LEGAL-BERT: The Muppets Straight Out of Law School. arXiv.10.18653/v1/2020.findings-emnlp.261 ↗
53. Huang, Y., Xu, J., Lai, J., Jiang, Z., Chen, T., Li, Z., and Zhao, P. (2023). Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey. arXiv.
54. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., and Raffel, C. (2020). mT5: A Massively Multilingual Pre-Trained Text-to-Text Transformer. arXiv.10.18653/v1/2021.naacl-main.41 ↗
55. Child, R., Gray, S., Radford, A., and Sutskever, I. (2019). Generating Long Sequences with Sparse Transformers. arXiv.
56. Ainslie, J., Ontanon, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., and Yang, L. (2020). ETC: Encoding Long and Structured Inputs in Transformers. arXiv.10.18653/v1/2020.emnlp-main.19 ↗
57. Zhang, H., Liu, X., and Zhang, J. (2022). Hegel: Hypergraph Transformer for Long Document Summarization. arXiv.10.18653/v1/2022.emnlp-main.692 ↗
58. He, H., Flicke, M., and Buchmann, J. (2024). HDT: Hierarchical Document Transformer. arXiv.
59. The Effects of Text Structure Instruction on Middle-Grade Students’ Comprehension and Production of Expository Text10.2307/747358 ↗
60. Roles of Document Structure, Cognitive Strategy, and Awareness in Searching for Information10.2307/747765 ↗
61. Murty, S., Sharma, P., and Andreas, J. (2023). Grokking of Hierarchical Structure in Vanilla Transformers. arXiv.10.18653/v1/2023.acl-short.38 ↗
62. Li, M., Hovy, E., and Lau, J.H. (2023). Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation. arXiv.10.18653/v1/2023.findings-emnlp.472 ↗
63. Cao, S., and Wang, L. (2022). HIBRIDS: Attention with Hierarchical Biases for Structure-Aware Long Document Summarization. arXiv.10.18653/v1/2022.acl-long.58 ↗
64. Ruan, Q., Ostendorff, M., and Rehm, G.H. (2022). Improving Extractive Text Summarization with Hierarchical Structure Information. arXiv.10.18653/v1/2022.findings-acl.102 ↗
65. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., and Stoyanov, V. (2019). Roberta: A Robustly Optimized BERT Pretraining Approach. arXiv.
66. Loshchilov, I., and Hutter, F. (2019, January 6–9). Decoupled Weight Decay Regularization. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA.
67. Xiong, W., Oğuz, B., Gupta, A., Chen, X., Liskovich, D., Levy, O., and Mehdad, Y. (2021). Simple Local Attentions Remain Competitive for Long-Context Tasks. arXiv.10.18653/v1/2022.naacl-main.144 ↗
68. Efficient Transformers: A Survey10.1145/3530811 ↗
69. Pham, H., Wang, G., Lu, Y., Florencio, D., and Zhang, C. (2022). Understanding Long Documents with Different Position-Aware Attentions. arXiv.
70. Tan, N.Ö., Peng, A.Y., Bensemann, J., Bao, Q., Hartill, T., Gahegan, M., and Witbrock, M. (2023). Input-Length-Shortening and Text Generation via Attention Values. arXiv.