SGR-Net: Learning Multimodal Embeddings for Traffic Accident Prediction via Geometry–State Attentive Fusion
Source abstract
Abstract
Traffic accident prediction is a key challenge in road safety, and it is necessary to accurately identify high-risk sections from different data sources. Although graphical neural networks (GNNs) simulate the road network topology well, they ignore the visual and environmental clues from physical road conditions. This paper addresses this gap by proposing a Sequential Geometric Reasoning Network (SGR-Net), a deep learning framework for multimodal accident prediction. Unlike prior GNN-based approaches, SGR-Net introduces a Geometry–State Attentive Fusion (GSAF) module—its main novelty—which dynamically integrates visual features from satellite imagery with structural graph contexts. The framework also includes a stability-aware training objective and meta-learning for cross-region generalization. We evaluate on a large-scale dataset covering six U.S. states with over nine million accidents and one million satellite images. SGR-Net achieves strong results, with AUROC up to 96.8% and MAE as low as 0.08 in Delaware. Ablations confirm the GSAF module is essential: removing it reduces AUROC by 2.7% and increases MAE by over 40%. The framework establishes a new state-of-the-art for multimodal traffic accident prediction.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Traffic and Road Safety · Automated Road and Building Extraction · Traffic Prediction and Management Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Macao Institute of Systems Engineering, Macau University of Science and Technology, Macau, China
Institute of Collaborative Innovation, University of Macao, Macao, China
Funding
Science and Technology Development Fund
References (65)
Show source-provided references
1. World Health Organization (2023). Global Status Report on Road Safety 2023, World Health Organization.
2. Kipf, T.N., and Welling, M. (2017, January 24–26). Semi-supervised classification with graph convolutional networks. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.
3. Hamilton, W.L., Ying, R., and Leskovec, J. (2017, January 4–9). Inductive representation learning on large graphs. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
4. Bai, L., Yao, L., Kanhere, S.S., Wang, X., Liu, W., and Yang, Z. (2019, January 3–7). Spatio-temporal graph convolutional and recurrent networks for citywide passenger demand prediction. Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM), Beijing, China.10.1145/3357384.3358097 ↗
5. Li, Y., Yu, R., Shahabi, C., and Liu, Y. (May, January 30). Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada.
6. Yu, B., Yin, H., and Zhu, Z. (2018, January 13–19). Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Stockholm, Sweden.10.24963/ijcai.2018/505 ↗
7. Interpretable machine learning for predicting cutting force of tunnel boring machine: A study on SVR with meta-heuristic optimization10.1002/nag.70083 ↗
8. Weakly supervised road network extraction for remote sensing image based scribble annotation and adversarial learning10.1016/j.jksuci.2022.05.020 ↗
9. Mnih, V., and Hinton, G.E. (2010, January 5–11). Learning to detect roads in high-resolution aerial images. Proceedings of the European Conference on Computer Vision (ECCV), Crete, Greece.10.1007/978-3-642-15567-3_16 ↗
10. He, S., Bastani, F., Jagwani, S., Park, E., Abbar, S., Alizadeh, M., Balakrishnan, H., Chawla, S., Madden, S., and Sadeghi, M.A. (2020, January 7–12). RoadTagger: Robust road attribute inference with graph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.10.1609/aaai.v34i07.6730 ↗
11. MG-TAR: Multi-view graph convolutional networks for traffic accident risk prediction10.1109/TITS.2023.3237072 ↗
12. Multimodal machine learning: A survey and taxonomy10.1109/TPAMI.2018.2798607 ↗
13. Tang, J., Yang, Y., Wei, W., Shi, L., Su, L., Cheng, S., Yin, D., and Huang, C. (2024, January 14–18). GraphGPT: Graph instruction tuning for large language models. Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, Washington, DC, USA.10.1145/3626772.3657775 ↗
14. Tang, J., Yang, Y., Wei, W., Shi, L., Xia, L., Yin, D., and Huang, C. (2024, January 25–29). HiGPT: Heterogeneous graph language model. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Barcelona, Spain.10.1145/3637528.3671987 ↗
15. Li, Z., Xia, L., Tang, J., Xu, Y., Shi, L., Xia, L., Yin, D., and Huang, C. (2024, January 25–29). UrbanGPT: Spatio-temporal large language models. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Barcelona, Spain.10.1145/3637528.3671578 ↗
16. Liao, R., Urtasun, R., and Zemel, R. (2021, January 3–7). A PAC-Bayesian approach to generalization bounds for graph neural networks. Proceedings of the International Conference on Learning Representations (ICLR), Virtual.
17. Ju, H., Li, D., Sharma, A., and Zhang, H.R. (2023, January 25–27). Generalization in graph neural networks: Improved PAC-Bayesian bounds on graph diffusion. Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), Valencia, Spain.
18. Li, D., Ju, H., Sharma, A., and Zhang, H.R. (2023, January 6–10). Boosting multitask learning on graphs through higher-order task affinities. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Long Beach, CA, USA.10.1145/3580305.3599265 ↗
19. Li, D., Sharma, A., and Zhang, H.R. (2024, January 25–29). Scalable multitask learning using gradient-based estimation of task affinity. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), Barcelona, Spain.10.1145/3637528.3671835 ↗
20. Li, D., Zhang, Z., Wang, L., and Zhang, H.R. (2024, January 12–16). Scalable fine-tuning from multiple data sources: A first-order approximation approach. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, FL, USA.10.18653/v1/2024.findings-emnlp.321 ↗
21. Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., and Askell, A. (2020, January 6–12). Language models are few-shot learners. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
22. Deep learning10.1038/nature14539 ↗
23. Wu, Z., Pan, S., Long, G., Jiang, J., and Zhang, C. (2019, January 10–16). Graph WaveNet for deep spatial-temporal graph modeling. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), Macao, China.10.24963/ijcai.2019/264 ↗
24. Cao, D., Jia, F., Arik, S.O., Pfister, T., Zheng, Y., Ye, W., and Liu, Y. (2024, January 7–11). TEMPO: Prompt-based generative pre-trained transformer for time series forecasting. Proceedings of the International Conference on Learning Representations (ICLR), Vienna, Austria.
25. He, S., Sadeghi, M.A., Chawla, S., Alizadeh, M., Balakrishnan, H., and Madden, S. (2021, January 10–17). Inferring high-resolution traffic accident risk maps based on satellite imagery and GPS trajectories. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01176 ↗
26. Multimodal learning with graphs10.1038/s42256-023-00624-6 ↗
27. DeepForest: A Python package for RGB deep learning tree crown delineation10.1111/2041-210X.13472 ↗
28. A review of solar forecasting, its dependence on atmospheric sciences and implications for grid integration: Towards carbon neutrality10.1016/j.rser.2022.112348 ↗
29. Máttyus, G., Luo, W., and Urtasun, R. (2017, January 22–29). DeepRoadMapper: Extracting road topology from aerial images. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.372 ↗
30. Predicting TBM penetration rate in hard rock condition: A comparative study among six XGB-based metaheuristic techniques10.1016/j.gsf.2020.09.020 ↗
31. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
32. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very deep convolutional networks for large-scale image recognition. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
33. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. (2009, January 20–25). ImageNet: A large-scale hierarchical image database. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Miami, FL, USA.10.1109/CVPR.2009.5206848 ↗
34. A comprehensive survey on graph neural networks10.1109/TNNLS.2020.2978386 ↗
35. Gilmer, J., Schoenholz, S.S., Riley, P.F., Vinyals, O., and Dahl, G.E. (2017, January 6–11). Neural message passing for quantum chemistry. Proceedings of the International Conference on Machine Learning (ICML), Sydney, NSW, Australia.
36. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural machine translation by jointly learning to align and translate. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
37. Lu, J., Yang, J., Batra, D., and Parikh, D. (2016, January 5–10). Hierarchical question-image co-attention for visual question answering. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain.
38. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.
39. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Y. (May, January 30). Graph attention networks. Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada.
40. Kendall, A., Gal, Y., and Cipolla, R. (2018, January 18–23). Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA.
41. Learning from imbalanced data10.1109/TKDE.2008.239 ↗
42. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.324 ↗
43. A survey on transfer learning10.1109/TKDE.2009.191 ↗
44. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. (2009, January 14–18). Curriculum learning. Proceedings of the International Conference on Machine Learning (ICML), Montreal, QC, Canada.10.1145/1553374.1553380 ↗
45. Finn, C., Abbeel, P., and Levine, S. (2017, January 6–11). Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the International Conference on Machine Learning (ICML), Sydney, NSW, Australia.
46. Nichol, A., Achiam, J., and Schulman, J. (2018). On first-order meta-learning algorithms. arXiv.
47. Kingma, D.P., and Ba, J. (2015, January 7–9). Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
48. Loshchilov, I., and Hutter, F. (2019, January 6–9). Decoupled weight decay regularization. Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA.
49. Dropout: A simple way to prevent neural networks from overfitting
50. Perozzi, B., Al-Rfou, R., and Skiena, S. (2014, January 24–27). DeepWalk: Online learning of social representations. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), New York, NY, USA.10.1145/2623330.2623732 ↗
51. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J. (2021, January 18–24). Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning (ICML), Virtual.
52. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., and Gelly, S. (2021, January 3–7). An image is worth 16 × 16 words: Transformers for image recognition at scale. Proceedings of the International Conference on Learning Representations (ICLR), Virtual.
53. Xu, K., Hu, W., Leskovec, J., and Jegelka, S. (2015, January 7–9). How powerful are graph neural networks?. Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA.
54. Ying, C., Cai, T., Luo, S., Zheng, S., Ke, G., He, D., Shen, Y., and Liu, T.-Y. (2021, January 6–14). Do transformers really perform badly for graph representation?. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Online.
55. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. (2020, January 6–12). Supervised contrastive learning. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
56. Gated multimodal networks10.1007/s00521-019-04559-1 ↗
57. Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. (2017, January 24–26). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Proceedings of the International Conference on Learning Representations (ICLR), Toulon, France.
58. The use of the area under the ROC curve in the evaluation of machine learning algorithms10.1016/S0031-3203(96)00142-2 ↗
59. Pearl, J. (2009). Causality: Models, Reasoning and Inference, Cambridge University Press. [2nd ed.].10.1017/CBO9780511803161 ↗
60. Causal inference using potential outcomes: Design, modeling, decisions10.1198/016214504000001880 ↗
61. Identification and estimation of local average treatment effects
62. Rosenbaum, P.R. (2002). Observational studies. Observational Studies, Springer.10.1007/978-1-4757-3692-2 ↗
63. Imbens, G.W., and Rubin, D.B. (2015). Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction, Cambridge University Press.10.1017/CBO9781139025751 ↗
64. Ribeiro, M.T., Singh, S., and Guestrin, C. (2016, January 13–17). “Why should I trust you?”: Explaining the predictions of any classifier. Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), San Francisco, CA, USA.10.1145/2939672.2939778 ↗
65. Lundberg, S.M., and Lee, S.-I. (2017, January 4–9). A unified approach to interpreting model predictions. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA.