A Review of Deep Learning-Based Visual Multi-Object Tracking Algorithms for Autonomous Driving
Source abstract
Abstract
Multi-target tracking, a high-level vision job in computer vision, is crucial to understanding autonomous driving surroundings. Numerous top-notch multi-object tracking algorithms have evolved in recent years as a result of deep learning’s outstanding performance in the field of visual object tracking. There have been a number of evaluations on individual sub-problems, but none that cover the challenges, datasets, and algorithms associated with visual multi-object tracking in autonomous driving scenarios. In this research, we present an exhaustive study of algorithms in the field of visual multi-object tracking over the last ten years, based on a systematic review approach. The algorithm is broken down into three groups based on its structure: methods for tracking by detection (TBD), joint detection and tracking (JDT), and Transformer-based tracking. The research reveals that the TBD algorithm has a straightforward structure, however the correlation between its individual sub-modules is not very strong. To track multiple objects, the JDT technique combines multi-module joint learning with a deep network framework. Transformer-based algorithms have been explored over the past two years, and they have benefits in numerous assessment indicators, as well as tremendous research potential in the area of multi-object tracking. Theoretical support for algorithmic research in adjacent disciplines is provided by this paper. Additionally, the approach we discuss, which uses merely monocular cameras rather than sophisticated sensor fusion, is anticipated to pave the way for the quick creation of safe and affordable autonomous driving systems.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Video Surveillance and Tracking Methods · Impact of Light on Environment and Health · Visual Attention and Saliency Detection
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Mechanical Engineering, North China University of Water Resources and Electric Power, Zhengzhou 457003, China
School of Intelligent Transportation, Yunnan Vocational College of Transportation, Kunming 650500, China
Funding
Ministry of education Industry–University Cooperation Education project · Key scientific research projects of colleges and universities in Henan Province · Scientific and Technological Innovation Team Support Program for Colleges and Universities in Henan Province · Key scientific research project plan of colleges and universities in Henan Province
References (93)
Show source-provided references
1. Preparing a nation for autonomous vehicles: Opportunities, barriers and policy recommendations10.1016/j.tra.2015.04.003 ↗
2. Autonomous cars: Research results, issues, and future challenges10.1109/COMST.2018.2869360 ↗
3. Leon, F., and Gavrilescu, M. (2019). A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving. arXiv.
4. Fan, L., Wang, Z., Cail, B., Tao, C., and Feng, Z. (2016, January 1–3). A survey on multiple object tracking algorithm. Proceedings of the 2016 IEEE International Conference on Information and Automation (ICIA), Ningbo, China.10.1109/ICInfA.2016.7832121 ↗
5. Luo, W., Xing, J., Zhang, X., Zhao, X., and Kim, T.K. (2014). Multiple Object Tracking: A Literature Review. arXiv.
6. Deep Learning in Video Multi-Object Tracking: A Survey10.1016/j.neucom.2019.11.023 ↗
7. A Survey of Multiple Pedestrian Tracking Based on Tracking-by-Detection Framework10.1109/TCSVT.2020.3009717 ↗
8. Krebs, S., Duraisamy, B., and Flohr, F. (2017, January 16–19). A survey on leveraging deep neural networks for object tracking. Proceedings of the 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan.10.1109/ITSC.2017.8317904 ↗
9. Deep learning for unmanned aerial vehicle-based object detection and tracking: A survey10.1109/MGRS.2021.3115137 ↗
10. Systematic literature reviews in software engineering–a systematic literature review10.1016/j.infsof.2008.09.009 ↗
11. Feichtenhofer, C., Pinz, A., and Zisserman, A. (2017, January 22–29). Detect to track and track to detect. Proceedings of the IEEE International Conference on Computer Vision (ICCV) 2017, Venice, Italy.10.1109/ICCV.2017.330 ↗
12. Fairmot: On the fairness of detection and re-identification in multiple object tracking10.1007/s11263-021-01513-4 ↗
13. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., and Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.
14. Wang, X., Girshick, R., Gupta, A., and He, K. (2017, January 21–26). Non-local Neural Networks. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2018.00813 ↗
15. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
16. Mnih, V., Heess, N., Graves, A., and Kavukcuoglu, K. (2014, January 8–13). Recurrent models of visual attention. Proceedings of the NIPS 2014, Montreal, QC, Canada.
17. Kim, C., Li, F., Ciptadi, A., and Rehg, J.M. (2015, January 7–13). Multiple hypothesis tracking revisited. Proceedings of the IEEE International Conference on Computer Vision 2015, Santiago, Chile.10.1109/ICCV.2015.533 ↗
18. Davey, S.J., Rutten, M.G., and Gordon, N.J. (2013). Track-before-detect techniques. Integrated Tracking, Classification, and Sensor Management, Wiley Online Library.10.1002/9781118450550.ch08 ↗
19. Reliable re-detection for long-term tracking10.1109/TCSVT.2018.2816570 ↗
20. Pang, B., Li, Y., Zhang, Y., Li, M., and Lu, C. (2020, January 13–19). Tubetk: Adopting tubes to track multi-object in a one-step training model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.00634 ↗
21. Multi-object tracking by joint detection and identification learning10.1007/s11063-019-10046-4 ↗
22. A model-based joint detection and tracking approach for multi-vehicle tracking with lidar sensor10.1109/TITS.2015.2391131 ↗
23. Zeng, F., Dong, B., Wang, T., Zhang, X., and Wei, Y. (2021). Motr: End-to-end multiple-object tracking with transformer. arXiv.10.1007/978-3-031-19812-0_38 ↗
24. Yu, E., Li, Z., Han, S., and Wang, H. Relationtrack: Relation-aware multiple object tracking with decoupled representation. IEEE Trans. Multimed., 2022.10.1109/TMM.2022.3150169 ↗
25. Milan, A., Leal-Taixe, L., Reid, I., Roth, S., and Schindler, K. (2016). MOT16: A Benchmark for Multi-Object Tracking. arXiv.
26. Dendorfer, P., Rezatofighi, H., Milan, A., Shi, J., Cremers, D., Reid, I., Roth, S., Schindler, K., and Leal-Taixé, L. (2020). MOT20: A benchmark for multi object tracking in crowded scenes. arXiv.
27. Geiger, A., Lenz, P., and Urtasun, R. (2012, January 16–21). Are we ready for autonomous driving? The KITTI vision benchmark suite. Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition, Providence, RI, USA.10.1109/CVPR.2012.6248074 ↗
28. Vision meets robotics: The KITTI dataset10.1177/0278364913491297 ↗
29. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020, January 13–19). nuscenes: A multimodal dataset for autonomous driving. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01164 ↗
30. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., and Caine, B. (2020, January 13–19). Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00252 ↗
31. Ertler, C., Mislej, J., Ollmann, T., Porzi, L., Neuhold, G., and Kuang, Y. (2020, January 23–28). The mapillary traffic sign dataset for detection and classification on a global scale. Proceedings of the ECCV 2020, Glasgow, UK.10.1007/978-3-030-58592-1_5 ↗
32. Leal-Taixé, L., Milan, A., Reid, I., Roth, S., and Schindler, K. (2015). Motchallenge 2015: Towards a benchmark for multi-target tracking. arXiv.
33. Evaluating multiple object tracking performance: The CLEAR MOT metrics
34. Hota: A higher order metric for evaluating multi-object tracking10.1007/s11263-020-01375-2 ↗
35. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
36. Distinctive Image Features from Scale-Invariant Keypoints10.1023/B:VISI.0000029664.99615.94 ↗
37. Dalal, N. (2005, January 20–26). Histograms of oriented gradients for human detection. Proceedings of the Computer Vision and Pattern Recognition, San Diego, CA, USA.
38. Dai, J., Li, Y., He, K., and Sun, J. (2016). R-FCN: Object Detection via Region-based Fully Convolutional Networks. arXiv.
39. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016). SSD: Single shot multibox detector. Lecture Notes in Computer Science, Springer.10.1007/978-3-319-46448-0_2 ↗
40. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
41. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
42. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, Faster, Stronger. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
43. Redmon, J., and Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv.
44. Bochkovskiy, A., Wang, C.Y., and Liao, H. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv.
45. Sonar tracking of multiple targets using joint probabilistic data association10.1109/JOE.1983.1145560 ↗
46. Multiple hypothesis tracking for multiple target tracking10.1109/MAES.2004.1263228 ↗
47. Jin, S., Ma, X., Han, Z., Wu, Y., Yang, W., Liu, W., Qian, C., and Ouyang, W. (2017, January 22–29). Towards multi-person pose tracking: Bottom-up and top-down methods. Proceedings of the ICCV PoseTrack Workshop, Venice, Italy.
48. Zhao, D., Fu, H., Xiao, L., Wu, T., and Dai, B. (2018). Multi-Object Tracking with Correlation Filter for Autonomous Vehicle. Sensors, 18.10.3390/s18072004 ↗
49. Li, T., Xu, D., Ma, Y., and Yu, C. (2018, January 13–15). A multiple object tracking algorithm based on YOLO detection. Proceedings of the 2018 11th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Beijing, China.
50. Bewley, A., Ge, Z., Ott, L., Ramos, F., and Upcroft, B. (2016, January 25–28). Simple online and realtime tracking. Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA.10.1109/ICIP.2016.7533003 ↗
51. Wojke, N., Bewley, A., and Paulus, D. (2017, January 17–20). Simple online and realtime tracking with a deep association metric. Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China.10.1109/ICIP.2017.8296962 ↗
52. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
53. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015, January 7–12). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298594 ↗
54. Xiao, T., Li, H., Ouyang, W., and Wang, X. (2016, January 27–30). Learning deep feature representations with domain guided dropout for person re-identification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.140 ↗
55. Multi-target tracking using CNN-based features: CNNMTT10.1007/s11042-018-6467-6 ↗
56. Fang, K., Xiang, Y., Li, X., and Savarese, S. (2018, January 12–15). Recurrent autoregressive networks for online multi-object tracking. Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA.10.1109/WACV.2018.00057 ↗
57. Heterogeneous association graph fusion for target association in multiple object tracking10.1109/TCSVT.2018.2882192 ↗
58. On the detection-to-track association for online multi-object tracking10.1016/j.patrec.2021.03.022 ↗
59. Du, Y., Song, Y., Yang, B., and Zhao, Y. (2022). StrongSORT: Make DeepSORT Great Again. arXiv.10.1109/TMM.2023.3240881 ↗
60. Son, J., Baek, M., Cho, M., and Han, B. (2017, January 21–26). Multi-object tracking with quadruplet convolutional neural networks. Proceedings of the Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.403 ↗
61. Multiple objects tracking by a highly decisive three-frame differencing-combined-background subtraction method with GMPFM-GMPHD filters and VGG16-LSTM classifier10.1016/j.jvcir.2020.102905 ↗
62. Online multi-object tracking based on feature representation and Bayesian filtering within a deep learning architecture10.1109/ACCESS.2019.2901520 ↗
63. LSTM network with tracking association for multi-object tracking
64. Milan, A., Rezatofighi, S.H., Dick, A.R., Reid, I.D., and Schindler, K. (2017). Online multi-target tracking using recurrent neural networks. Proc. Conf. AAAI Artif. Intell., 31.10.1609/aaai.v31i1.11194 ↗
65. Kim, C., Li, F., and Rehg, J.M. (2018, January 8–14). Multi-object tracking with neural gating using bilinear lstm. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01237-3_13 ↗
66. Ran, N., Kong, L., Wang, Y., and Liu, Q. (2019, January 8–11). A robust multi-athlete tracking algorithm by exploiting discriminant features and long-term dependencies. Proceedings of the International Conference on Multimedia Modeling, Thessaloniki, Greece.10.1007/978-3-030-05710-7_34 ↗
67. Lee, B., Erdenee, E., Jin, S., Nam, M.Y., Jung, Y.G., and Rhee, P.K. (October, January 8–10,). Multi-class multi-object tracking using changing point detection. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.
68. Kim, C., Li, F., Alotaibi, M., and Rehg, J.M. (2021, January 20–25). Discriminative Appearance Modeling with Multi-track Pooling for Real-time Multi-object Tracking. Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA.10.1109/CVPR46437.2021.00943 ↗
69. Cao, J., Weng, X., Khirodkar, R., Pang, J., and Kitani, K. (2022). Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking. arXiv.10.1109/CVPR52729.2023.00934 ↗
70. Bergmann, P., Meinhardt, T., and Leal-Taixe, L. (2019–02, January 27). Tracking without bells and whistles. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea.10.1109/ICCV.2019.00103 ↗
71. Zhang, J., Zhou, S., Chang, X., Wan, F., and Huang, D. (2020). Multiple Object Tracking by Flowing and Fusing. arXiv.
72. Huang, P., Han, S., Zhao, J., Liu, D., Wang, H., Yu, E., and Kot, C.C. (2020). Refinements in Motion and Appearance for Online Multi-Object Tracking. arXiv.
73. Zhou, X., Koltun, V., and Krhenbühl, P. (2020). Tracking Objects as Points. arXiv.10.1007/978-3-030-58548-8_28 ↗
74. Peng, J., Wang, C., Wan, F., Wu, Y., and Fu, Y. (2020). Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking. arXiv.10.1007/978-3-030-58548-8_9 ↗
75. Wang, Z., Zheng, L., Liu, Y., Li, Y., and Wang, S. (2020). Towards real-time multi-object tracking. arXiv.10.1007/978-3-030-58621-8_7 ↗
76. Zhu, J., Yang, H., Liu, N., Kim, M., Zhang, W., and Yang, M.H. (2019). Online Multi-Object Tracking with Dual Matching Attention Networks. arXiv.10.1007/978-3-030-01228-1_23 ↗
77. Feng, W., Hu, Z., Wei, W., Yan, J., and Ouyang, W. (2019). Multi-Object Tracking with Multiple Cues and Switcher-Aware Classification. arXiv.
78. Chu, P., Fan, H., Tan, C., and Ling, H. (2019, January 7–11). Online Multi-Object Tracking With Instance-Aware Tracker and Dynamic Model Refreshment. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV.2019.00023 ↗
79. Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural Machine Translation by Jointly Learning to Align and Translate. arXiv.
80. Imagenet classification with deep convolutional neural networks10.1145/3065386 ↗
81. Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., and Xu, Y. (2020). A survey on visual transformer. arXiv.
82. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. (2020). Deformable detr: Deformable transformers for end-to-end object detection. arXiv.
83. Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., and Lu, H. (2019, January 15–20). Dual Attention Network for Scene Segmentation. Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00326 ↗
84. Trseg: Transformer for semantic segmentation10.1016/j.patrec.2021.04.024 ↗
85. Lu, D., Xie, Q., Xu, L., and Li, J. (2022). 3DCTN: 3D Convolution-Transformer Network for Point Cloud Classification. arXiv.10.1109/TITS.2022.3198836 ↗
86. Pct: Point cloud transformer10.1007/s41095-021-0229-5 ↗
87. Sun, P., Jiang, Y., Zhang, R., Xie, E., and Luo, P. (2020). TransTrack: Multiple-Object Tracking with Transformer. arXiv.
88. Meinhardt, T., Kirillov, A., Leal-Taixe, L., and Feichtenhofer, C. (2021). TrackFormer: Multi-Object Tracking with Transformers. arXiv.10.1109/CVPR52688.2022.00864 ↗
89. Xu, Y., Ban, Y., Delorme, G., Gan, C., Rus, D., and Alameda-Pineda, X. (2021). TransCenter: Transformers with Dense Queries for Multiple-Object Tracking. arXiv.
90. Chu, P., Wang, J., You, Q., Ling, H., and Liu, Z. (2021). TransMOT: Spatial-Temporal Graph Transformer for Multiple Object Tracking. arXiv.
91. Xie, F., Wang, C., Wang, G., Yang, W., and Zeng, W. (2021). Learning Tracking Representations via Dual-Branch Fully Transformer Networks. arXiv.10.1109/ICCVW54120.2021.00303 ↗
92. Zhu, X., Jia, Y., Jian, S., Gu, L., and Pu, Z. (2021). ViTT: Vision Transformer Tracker. Sensors, 21.10.3390/s21165608 ↗
93. Transformer-based two-source motion model for multi-object tracking10.1007/s10489-021-03012-y ↗