Dual-Linear Attention Network for Multi-Object Tracking and Segmentation
Source abstract
Abstract
Multi-object tracking and segmentation (MOTS) is a critical task in video analysis with applications spanning autonomous driving, robot navigation, and scene understanding. MOTS has made significant progress but still faces persistent challenges, such as crowded scenes, abnormal illumination, and small objects. Several trackers have implemented attention mechanisms to overcome these difficulties. However, many attention mechanisms have quadratic computational complexity and use little spatio-temporal information. This paper proposes a Dual-Linear Attention Network (DLAN), a novel approach that effectively integrates both appearance and spatio-temporal information while maintaining linear attention complexity. DLAN employs recursive linear self-attention to strengthen the appearance representation and prototypical linear cross-attention to condense rich spatio-temporal information, which can compensate for missing pixel information. DLAN optimizes both image features and segmentation, with the refined segmentation guiding frame-level memory updates to improve instance consistency. Extensive experiments on BDD100K MOT, BDD100K MOTS, and KITTI MOTS datasets demonstrate the following: (1) The three main challenges of object occlusion, illumination variation, and distant objects have been successfully mitigated by integrating DLAN. (2) DLAN has achieved an overall competitive performance when compared to state-of-the-art trackers, with a 26% reduction in identity switches (IDS) when compared to QDTrack-mots-fix.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Advanced Neural Network Applications · Human Pose and Action Recognition
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
College of Automation, Jiangsu University of Science and Technology, Zhenjiang 212100, China
College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China
References (59)
Show source-provided references
1. Segment as Points for Efficient and Effective Online Multi-Object Tracking and Segmentation10.1109/TPAMI.2021.3087898 ↗
2. Voigtlaender, P., Krause, M., Osep, A., Luiten, J., Sekar, B.B.G., Geiger, A., and Leibe, B. (2019, January 15–21). Mots: Multi-Object Tracking and Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00813 ↗
3. Sheng, H., Zhang, D., Yang, D., Yang, D., Liu, X., and Ke, W. (2025). Occlusion-Robust Multi-Target Tracking and Segmentation Framework with Mask Enhancement. Appl. Sci., 15.10.3390/app15136969 ↗
4. SCATT: Transformer Tracking with Symmetric Cross-Attention10.1007/s10489-024-05467-1 ↗
5. LUXFormer: Low-Light Image Enhancement via Joint Spatial-Frequency Illumination Modeling10.1007/s44443-025-00270-5 ↗
6. Zhou, J., Pang, Z., and Wang, Y. (2024, January 17–21). RMem: Restricted Memory Banks Improve Video Object Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR52733.2024.01760 ↗
7. Keles, F.D., Wijewardena, P.M., and Hegde, C. (2023, January 20–23). On the Computational Complexity of Self-Attention. Proceedings of the 34th International Conference on Algorithmic Learning Theory, Singapore.
8. Yan, B., Peng, H., Fu, J., Wang, D., and Lu, H. (2021, January 11–17). Learning Spatio-Temporal Transformer for Visual Tracking. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada.10.1109/ICCV48922.2021.01028 ↗
9. STMT: Spatio-Temporal Memory Transformer for Multi-Object Tracking10.1007/s10489-023-04617-1 ↗
10. Lovit: Long VideoTransformer for Surgical Phase Recognition10.1016/j.media.2024.103366 ↗
11. Ke, L., Li, X., Danelljan, M., Tai, Y., Tang, C., and Yu, F. (2021, January 6–14). Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation. Proceedings of the Advances in Neural Information Processing Systems 34 (NeurIPS), Virtual.
12. Lu, X., Diao, W., Mao, Y., Li, J., Wang, P., Sun, X., and Fu, K. (2023, January 7–14). Breaking Immutable: Information-Coupled Prototype Elaboration for Few-Shot Object Detection. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Washington, DC, USA.10.1609/aaai.v37i2.25274 ↗
13. Wang, S., Li, B.Z., Khabsa, M., Fang, H., and Ma, H. (2020). Linformer: Self-Attention with Linear Complexity. arXiv.
14. Fan, Q., Huang, H., and He, R. (2025, January 11–15). Breaking the Low-Rank Dilemma of Linear Attention. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR52734.2025.02353 ↗
15. Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. (2020, January 13–19). BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00271 ↗
16. Xu, Z., Zhang, W., Tan, X., Yang, W., Su, X., Yuan, Y., Zhang, H., Wen, S., Ding, E., and Huang, L. (2020). Pointtrack++ for Effective Online Multi-Object Tracking and Segmentation. arXiv.10.1007/978-3-030-58452-8_16 ↗
17. SiamMask: A Framework for Fast Online Object Tracking and Segmentation
18. MaskTrack: Auto-Labeling and Stable Tracking for Video Object Segmentation10.1109/TNNLS.2024.3469959 ↗
19. Interactive Multi-Scale Fusion of 2D and 3D Features for Multi-Object Vehicle Tracking10.1109/TITS.2023.3275954 ↗
20. FFFN: Frame-By-Frame Feedback Fusion Network for Video Super-Resolution10.1109/TMM.2022.3214776 ↗
21. SiamMaskAttn: Inverted Residual Attention Block Fusing Multi-Scale Feature Information for Multitask Visual Object Tracking Networks10.1007/s11760-023-02827-1 ↗
22. Detection and Tracking of Floating Objects Based on Spatial-Temporal Information Fusion10.1016/j.eswa.2023.120185 ↗
23. Sun, P., Cao, J., Jiang, Y., Zhang, R., Xie, E., Yuan, Z., Wang, C., and Luo, P. (2021). TransTrack: Multiple Object Tracking with Transformer. arXiv.
24. Multi-Object Tracking Based on Attention Networks for Smart City system
25. EANTrack: An Efficient Attention Network for Visual Tracking10.1109/TASE.2023.3319676 ↗
26. Xiao, C., Cao, Q., Zhong, Y., Zhang, X., Wang, T., and Yang, C. (2025). Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, IEEE.10.1109/TCSVT.2025.3615851 ↗
27. EAPT: Efficient Attention Pyramid Transformer for Image Processing10.1109/TMM.2021.3120873 ↗
28. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
29. Lin, T., Dollar, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature Pyramid Networks for Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
30. He, K., Gkioxari, G., Dollar, P., and Girshick, R. (2017, January 22–29). Mask R-CNN. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.322 ↗
31. QDTrack: Quasi-Dense Similarity Learning for Appearance-Only Multiple Object Tracking10.1109/TPAMI.2023.3301975 ↗
32. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems 30 (NeurIPS), Long Beach, CA, USA.
33. Transformer-Based Assignment Decision Network for Multiple Object Tracking10.1016/j.cviu.2024.103957 ↗
34. MOT-DETR: 3D Single Shot Detection and Tracking with Transformers to Build 3D Representations for Agro-Food Robots10.1016/j.compag.2024.109275 ↗
35. Clevert, D., Unterthiner, T., and Hochreiter, S. (2016, January 2–4). Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). International Conference on Learning Representations (ICLP), San Juan, Puerto Rico.
36. Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. (2020, January 13–18). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. Proceedings of the 37th International Conference on Machine Learning, Virtual.
37. Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., and Liu, H. (November, January 27). Expectation-Maximization Attention Networks for Semantic Segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
38. Entangled Appearance and Motion Structures Network for Multi-Object Tracking and Segmentation10.1007/s00138-024-01634-z ↗
39. Towards High Quality Multi-Object Tracking and Segmentation without Mask Supervision10.1109/TIP.2024.3403497 ↗
40. Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation
41. Yan, B., Jiang, Y., Wu, J., Wang, D., Luo, P., Yuan, Z., and Lu, H. (2023, January 18–22). Universal Instance Perception as Object Discovery and Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.01471 ↗
42. Ke, L., Danelljan, M., Ding, H., Tai, Y., Tang, C., and Yu, F. (2023, January 18–22). Mask-Free Video Instance Segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.02189 ↗
43. Zhang, Y., Wang, T., and Zhang, X. (2023, January 18–22). MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada.10.1109/CVPR52729.2023.02112 ↗
44. Zeng, F., Dong, B., Zhang, Y., Wang, T., Zhang, X., and Wei, Y. (2022, January 23–27). MOTR: End-to-End Multiple-Object Tracking with Transformer. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19812-0_38 ↗
45. Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F., Yuan, Z., Luo, P., Liu, W., and Wang, X. (2022, January 23–27). ByteTrack: Multi-object Tracking by Associating Every Detection Box. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-20047-2_1 ↗
46. Ke, L., Ding, H., Danelljan, M., Tai, Y., Tang, C., and Yu, F. (2022, January 23–27). Video Mask Transfiner for High-Quality Video Instance Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1109/CVPR52688.2022.00437 ↗
47. Yan, B., Jiang, Y., Sun, P., Wang, D., Yuan, Z., Luo, P., and Lu, H. (2022, January 23–27). Towards Grand Unification of Object Tracking. Proceedings of the European Conference on Computer Vision (ECCV), Tel Aviv, Israel.10.1007/978-3-031-19803-8_43 ↗
48. Cui, Y., Cao, Z., Xie, Y., Jiang, X., Tao, F., Chen, Y.V., Li, L., and Liu, D. (2022, January 3–8). DG-Labeler and DGL-MOTS Dataset: Boost the Autonomous Driving Perception. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV51458.2022.00347 ↗
49. Cai, J., Wang, Y., Hsu, H., Zhang, H., and Hwang, J. (2022, January 4–8). DIOR: DIstill Observations to Representations for Multi-Object Tracking and Segmentation. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Waikoloa, HI, USA.10.1109/WACVW54805.2022.00058 ↗
50. Yang, F., Chang, X., Dang, C., Zheng, Z., Sakti, S., Nakamura, S., and Wu, Y. (2021). ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation. arXiv.
51. Unde, A.S., and Rameshan, R.M. (2021). MOTS R-CNN: Cosine-Margin-Triplet Loss for Multi-Object Tracking. arXiv.
52. Porzi, L., Hofinger, M., Ruiz, I., Serrat, J., Bulo, S.R., and Kontschieder, P. (2020, January 13–19). Learning Multi-Object Tracking and Segmentation from Automatic Annotations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA.10.1109/CVPR42600.2020.00688 ↗
53. Xu, Z., Zhang, W., Tan, X., Yang, W., Huang, H., Wen, S., Ding, E., and Huang, L. (2020, January 23–28). Segment as Points for Efficient Online Multi-Object Tracking and Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58452-8_16 ↗
54. Athar, A., Mahadevan, S., Osep, A., Leal-Taixé, L., and Leibe, B. (2020, January 23–28). STEm-Seg: Spatio-Temporal Embeddings for Instance Segmentation in Videos. Proceedings of the European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58621-8_10 ↗
55. Track to Reconstruct and Reconstruct to Track10.1109/LRA.2020.2969183 ↗
56. Zhou, X., Koltun, V., and Krähenbühl, P. (2020, January 23–28). Tracking Objects as Points. European Conference on Computer Vision (ECCV), Glasgow, UK.10.1007/978-3-030-58548-8_28 ↗
57. Yang, L., Fan, Y., and Xu, N. (November, January 27). Video Instance Segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Republic of Korea.
58. Sharma, S., Ansari, J.A., Murthy, J.K., and Krishna, K.M. (2018, January 21–25). Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, QLD, Australia.10.1109/ICRA.2018.8461018 ↗
59. Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B. (2016, January 25–28). Simple Online and Realtime Tracking. Proceedings of the IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7533003 ↗