Cross-Modal Behavioral Intelligence in Regard to a Ship Bridge: A Rough Set-Driven Framework with Enhanced Spatiotemporal Perception and Object Semantics

Aberrant or non-standard operations by ship drivers are a leading cause of water traffic accidents, making the development of real-time and reliable behavior detection systems critically important. However, the environment within a ship’s bridge is significantly more complex than typical scenarios, such as vehicle driving or general security monitoring, which results in poor performance when applying generic algorithms. In such settings, both the accuracy and efficiency of existing methods are notably limited. To address these challenges, this paper proposes a cross-modal behavioral intelligence framework designed specifically for a ship’s bridge, integrating multi-target tracking, behavior recognition, and feature object association. The framework employs ByteTrack, a high-performance multi-object tracker that maintains stable tracking even when subject to occlusions or motion blur through its novel association mechanism, using both high and low confidence detection boxes, for multi-driver tracking. Combined with an improved Temporal Shift Module (TSM) algorithm for behavior recognition, which effectively resolves issues concerning target association and action ambiguity in complex environments, the proposed framework achieves a Top-1 accuracy of 82.1%, based on the SCA dataset. Furthermore, the method incorporates a multi-modal decision optimization strategy, based on spatiotemporal correlation rules, leveraging YOLOv7-e6 for simultaneous personnel and small object detection, and introduces the Accuracy of Focused Anomaly Recognition (AFAR) metric to enhance the anomaly detection performance. This approach improves the anomaly detection rate, up to 81.37%, with an overall accuracy of 80.66%, significantly outperforming single-modality solutions.