Enhancing Port Security: A Dual-Stage Multimodal Agent for Reducing False Alarms in Human–Vehicle Interaction Detection
Source abstract
Abstract
In the field of port security, traditional human–vehicle interaction conflict (HVIC) alarm algorithms predominantly rely on the bounding box overlap ratio. This criterion often fails in complex industrial environments, leading to excessive false positives caused by stationary vehicles, perspective distortion, and boarding/alighting activities. To address this limitation, this study proposes a dual-stage intelligent agent architecture designed to minimize false alarms. The system integrates YOLOv8 as a front-end lightweight detector for real-time candidate screening and Qwen2.5-VL, a domain-adaptive multimodal large model, as the back-end semantic verifier. A comprehensive dataset comprising one million port-specific images and videos was curated to support a novel two-phase training strategy: image pretraining for object recognition followed by video fine-tuning for temporal logic understanding. The agent dynamically interprets alarm events within their spatiotemporal context. Field trials at an operational wharf demonstrate that the proposed agent achieves an alarm precision of 95.7% and reduces false positives by over 50% across major error categories. This approach offers a highly reliable, automated solution for industrial security monitoring.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Maritime Navigation and Safety · Advanced Neural Network Applications · Anomaly Detection Techniques and Applications
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Zhejiang University-University of Illinois Urbana-Champaign Institute, Zhejiang University, No. 718 Haizhou East Road, Haining 314400, China
Zhejiang Provincial Seaport Investment & Operation Group Co., Ltd., Ningbo 315042, China
References (64)
Show source-provided references
1. Trimble, D., Monken, J., and Sand, A. (2017, January 7–8). A framework for cybersecurity assessments of critical port infrastructure. Proceedings of the 2017 International Conference on Cyber Conflict (CyCon U.S.), Washington, DC, USA.10.1109/CYCONUS.2017.8167506 ↗
2. Giovannetti, A., Gotelli, M., Solina, V., and Tonelli, F. (2025). Assessing Port Facility Safety: A Comparative Analysis of Global Accident and Injury Databases. Appl. Sci., 15.10.3390/app152211961 ↗
3. Smart ports for sustainable shipping: Concept and practices revisited through the case study of China’s Tianjin port10.1057/s41278-024-00291-3 ↗
4. Zou, Y., Xiao, G., Li, Q., and Biancardo, S. (2025). Intelligent Maritime Shipping: A Bibliometric Analysis of Internet Technologies and Automated Port Infrastructure Applications. J. Mar. Sci. Eng., 13.10.3390/jmse13050979 ↗
5. Wang, H., Wang, H., and Tang, X. (2025). A Review of Deep Learning in Rotating Machinery Fault Diagnosis and Its Prospects for Port Applications. Appl. Sci., 15.10.3390/app152111303 ↗
6. CBC: Caching for cloud-based VOD systems10.1007/s11042-013-1663-x ↗
7. Issa Zadeh, S.B., Esteban Perez, M.D., López-Gutiérrez, J.S., and Fernández-Sánchez, G. (2023). Optimizing Smart Energy Infrastructure in Smart Ports: A Systematic Scoping Review of Carbon Footprint Reduction. J. Mar. Sci. Eng., 11.10.3390/jmse11101921 ↗
8. Paias, A., Ruthmair, M., and Voss, S. (2016). Agent-Based Support for Container Terminals to Make Appointments with Barges. Computational Logistics, Springer International Publishing.10.1007/978-3-319-44896-1 ↗
9. The impact of neurotechnology on maritime port security—Hypothetical port10.1007/s12198-022-00253-x ↗
10. Lee, J.M., Sim, M.S., Kim, Y.S., Lim, H.R., and Lee, C.H. (2025). Strategizing Artificial Intelligence Transformation in Smart Ports: Lessons from Busan’s Resilient AI Governance Model. J. Mar. Sci. Eng., 13.10.3390/jmse13071276 ↗
11. Sustainability practices for SDGs: A study of Brazilian ports10.1007/s10668-023-03126-y ↗
12. A framework for building a smart port and smart port index10.1080/15568318.2019.1610919 ↗
13. Wang, Y., Song, Z., and Zhang, L. (2024). Detection of Safety Signs Using Computer Vision Based on Deep Learning. Appl. Sci., 14.10.3390/app14062556 ↗
14. Analyzing Real-Time Object Detection with YOLO Algorithm in Automotive Applications: A Review
15. Motion-Aware Tiny Object Detection using YOLO-MoNet10.1016/j.procs.2025.04.584 ↗
16. Tavakoli, H., Suh, S., Walunj, S., Pahlevannejad, P., Plociennik, C., and Ruskowski, M. (2023). Object Detection for Human—Robot Interaction and Worker Assistance Systems. Artificial Intelligence in Manufacturing: Enabling Intelligent, Flexible and Cost-Effective Production Through AI, Springer Nature Switzerland.10.1007/978-3-031-46452-2_18 ↗
17. A YOLOv8 algorithm for safety helmet wearing detection in complex environment10.1038/s41598-025-08828-z ↗
18. Deniz, J.M., Kelboucas, A.S., and Grando, R.B. (2024, January 11–14). Real-time robotics situation awareness for accident prevention in industry. Proceedings of the 2024 Latin American Robotics Symposium (LARS), Arequipa, Peru.10.1109/LARS64411.2024.10786413 ↗
19. Genevois, T., Spalanzani, A., and Laugier, C. (2023, January 4–7). Interaction-aware predictive collision detector for human-aware collision avoidance. Proceedings of the 2023 IEEE Intelligent Vehicles Symposium (IV), Anchorage, AK, USA.10.1109/IV55152.2023.10186778 ↗
20. A YOLO-Based Intelligent Detection Algorithm for Risk Assess-Ment of Construction Sites10.26599/Jic.2024.9180037 ↗
21. Cheng, X., Wang, F., Hu, X., Wu, X., and Nuo, M. (2025). IRWT-YOLO: A Background Subtraction-Based Method for Anti-Drone Detection. Drones, 9.10.3390/drones9040297 ↗
22. YoloGA: An Evolutionary Computation Based YOLO Algorithm to Detect Personal Protective Equipment
23. Elmir, Y., Touati, H., and Melizou, O. (2024). Intelligent Video Recording Optimization using Activity Detection for Surveillance Systems. arXiv.
24. YOLOv8-BS: An integrated method for identifying stationary and moving behaviors of cattle with a newly developed dataset10.1016/j.atech.2025.101153 ↗
25. Unsafe-Net: YOLO v4 and ConvLSTM based computer vision system for real-time detection of unsafe behaviours in workplace10.1007/s11042-024-19276-8 ↗
26. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. arXiv.
27. Development Status and Strategy Analysis of Medical Big Models
28. McCoy, L.G., Ng, F.Y.C., Sauer, C.M., Legaspi, K.E.Y., Jain, B., Gallifant, J., McClurkin, M., Hammond, A., Goode, D., and Gichoya, J. (2024). Understanding and training for the impact of large language models and artificial intelligence in healthcare practice: A narrative review. Bmc Med. Educ., 24.10.1186/s12909-024-06048-z ↗
29. OpenAI (2023). GPT-4 Technical Report, OpenAI. Technical Report.
30. Wu, Z., Chen, X., Pan, Z., Liu, X., Liu, W., Dai, D., Gao, H., Ma, Y., Wu, C., and Wang, B. (2024). DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding. arXiv.
31. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv.
32. Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., and Tang, J. (2025). Qwen2.5-VL Technical Report. arXiv.
33. Liu, Z., Huang, D., Huang, K., Li, Z., and Zhao, J. (2020, January 11–17). FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining. Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI-20), Yokohama, Japan.10.24963/ijcai.2020/622 ↗
34. Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. (2023). BloombergGPT: A Large Language Model for Finance. arXiv.
35. Vaghefi, S.A., Huggel, C., Muccione, V., Khashehchi, H., and Leippold, M. (2022, January 9). Deep Climate Change: A Dataset and Adaptive domain pre-trained Language Models for Climate Change Related Tasks. Proceedings of the NeurIPS 2022 Workshop on Tackling Climate Change with Machine Learning, Online.
36. ChatClimate: Grounding conversational AI in climate science10.1038/s43247-023-01084-x ↗
37. (2025, December 24). YOLOv8: The Latest Advancement in Real-Time Object Detection; Ultralytics. Available online: https://docs.ultralytics.com/models/yolov8/.
38. Yang, G., Wang, J., Nie, Z., Yang, H., and Yu, S. (2023). A lightweight YOLOv8 tomato detection algorithm combining feature enhancement and attention. Agronomy, 13.10.3390/agronomy13071824 ↗
39. Shi, J., Bai, Y., Zhou, J., and Zhang, B. (2023). Multi-crop navigation line extraction based on improved YOLO-v8 and threshold-DBSCAN under complex agricultural environments. Agriculture, 14.10.3390/agriculture14010045 ↗
40. Chi, X., Sun, Y., Zhao, Y., Lu, D., Gao, Y., and Zhang, Y. (2024). An Improved YOLOv8 network for detecting electric pylons based on optical satellite image. Sensors, 24.10.3390/s24124012 ↗
41. Song, X., Cao, S., Zhang, J., and Hou, Z. (2024). Steel surface defect detection algorithm based on YOLOv8. Electronics, 13.10.3390/electronics13050988 ↗
42. Xiao, Y., Yang, H., Dai, D., Wang, H., Shan, Z., and Wu, H. (2025). CKAN-YOLOv8: A Lightweight Multi-Task Network for Underwater Target Detection and Segmentation in Side-Scan Sonar. J. Mar. Sci. Eng., 13.10.20944/preprints202504.0701.v1 ↗
43. Factors influencing unsafe behaviors and accidents on construction sites: A review10.1080/10803548.2014.11077023 ↗
44. Chen, B., Wang, X., Bao, Q., Jia, B., Li, X., and Wang, Y. (2022). An unsafe behavior detection method based on improved YOLO framework. Electronics, 11.10.3390/electronics11121912 ↗
45. Liu, Y., Wang, P., and Li, H. (2025). An improved yolov5s-based algorithm for unsafe behavior detection of construction workers in construction scenarios. Appl. Sci., 15.10.3390/app15041853 ↗
46. A novel deep learning framework for detecting seafarer’s unsafe behavior
47. Clarifying Unsafe Behaviors of Construction Workers through a Complex Network of Unsafe Behavior Chains10.1061/JCEMD4.COENG-14956 ↗
48. An experimental study of real-time identification of construction workers’ unsafe behaviors10.1016/j.autcon.2017.05.002 ↗
49. Safety leadership and safety behavior in container terminal operations10.1016/j.ssci.2009.05.003 ↗
50. Exploring the effect of timely reminder on maritime unsafe acts10.1177/0361198120925460 ↗
51. Wukak, M.M.U., Berek, N.C., Sahdan, M., and Nabuasa, D.J. (2021, January 29–30). Unsafe Behavior on Dockworkers at Lewoleba Sea Port. Proceedings of the The International Conference on Public Health Proceeding, Solo, Indonesia.
52. An analysis of factors associated with the safety behavior of ship inspection employees safety in port health office class I Surabaya10.5958/0976-5506.2018.00427.8 ↗
53. Improvement of Unsafe Act and Unsafe Condition Online Reporting System at Port
54. AutoCon: Large Vision–Language Models for Automatic Construction Hazard Assessment
55. Spurnỳ, T., Babjak, J., Bobovskỳ, Z., and Vysockỳ, A. (2025). Dynamic Proxemic Model for Human–Robot Interactions Using the Golden Ratio. Appl. Sci., 15.10.3390/app15158130 ↗
56. Abdullah, A., Rahman, M., and Yan, X. (2025, January 11–15). ISafetyBench: A Video–Language Benchmark for Industrial Safety Monitoring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.
57. Efficient Multimodal Reasoning via Lightweight Vision–Language Architectures
58. International Labour Office (2005). Safety and Health in Ports: ILO Code of Practice, International Labour Organization.
59. (2024). Safety of Machinery: Positioning of Safeguards with Respect to the Approach Speeds of Parts of the Human Body (Standard No. ISO 13855:2024).
60. Advancing Human-Robot Collaboration: A Focus on Speed and Separation Monitoring10.4236/ojapps.2025.154060 ↗
61. (2020). Industrial Trucks: Safety Requirements and Verification (Standard No. ISO 3691-1:2020).
62. Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction10.1016/j.aei.2021.101400 ↗
63. Learning Transferable Visual Models From Natural Language Supervision
64. Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv.