A Comprehensive Review of Computer Vision in Sports: Open Issues, Future Trends and Research Directions
Source abstract
Abstract
Recent developments in video analysis of sports and computer vision techniques have achieved significant improvements to enable a variety of critical operations. To provide enhanced information, such as detailed complex analysis in sports such as soccer, basketball, cricket, and badminton, studies have focused mainly on computer vision techniques employed to carry out different tasks. This paper presents a comprehensive review of sports video analysis for various applications: high-level analysis such as detection and classification of players, tracking players or balls in sports and predicting the trajectories of players or balls, recognizing the team’s strategies, and classifying various events in sports. The paper further discusses published works in a variety of application-specific tasks related to sports and the present researcher’s views regarding them. Since there is a wide research scope in sports for deploying computer vision techniques in various sports, some of the publicly available datasets related to a particular sport have been discussed. This paper reviews detailed discussion on some of the artificial intelligence (AI) applications, GPU-based work-stations and embedded platforms in sports vision. Finally, this review identifies the research directions, probable challenges, and future trends in the area of visual recognition in sports.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Human Pose and Action Recognition · Video Analysis and Summarization · Autonomous Vehicle Technology and Safety
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electronics and Communication Engineering, National Institute of Technology, Warangal 506004, India
Department of Civil and Architectural Engineering, Aarhus University, 8000 Aarhus, Denmark
References (100+)
Show source-provided references
1. Tan, D., Ting, H., and Lau, S. (2016, January 5–6). A review on badminton motion analysis. Proceedings of the International Conference on Robotics, Automation and Sciences (ICORAS), Melaka, Malaysia.10.1109/ICORAS.2016.7872604 ↗
2. Computational intelligence in sports: A systematic literature review10.1155/2018/3426178 ↗
3. A survey of video based action recognition in sports
4. Accuracy of human motion capture systems for sport applications; state-of-the-art review10.1080/17461391.2018.1463397 ↗
5. A survey on player tracking in soccer videos10.1016/j.cviu.2017.02.002 ↗
6. Computer vision for sports: Current applications and research topics10.1016/j.cviu.2017.04.011 ↗
7. Machine and deep learning for sport-specific movement recognition: A systematic review of model development and performance10.1080/02640414.2018.1521769 ↗
8. Ball tracking in sports: A survey10.1007/s10462-017-9582-2 ↗
9. A survey of content-aware video analysis for sports10.1109/TCSVT.2017.2655624 ↗
10. Artificial intelligence for team sports: A survey10.1017/S0269888919000225 ↗
11. Video Summarization Using Deep Neural Networks: A Survey10.1109/JPROC.2021.3117472 ↗
12. Adesida, Y., Papi, E., and McGregor, A.H. (2019). Exploring the role of wearable technology in sport kinematics and kinetics: A systematic review. Sensors, 19.10.3390/s19071597 ↗
13. Wearable sensors for real-time kinematics analysis in sports: A review10.1109/JSEN.2020.3019016 ↗
14. Real Time Moving Vehicle Congestion Detection and Tracking using OpenCV
15. Davis, M. (2008). Investigation into Tracking Football Players from Single Viewpoint Video Sequences. [Bachelor’s Thesis, The University of Bath].
16. Spagnolo, P., Mosca, N., Nitti, M., and Distante, A. (2007, January 5–7). An unsupervised approach for segmentation and clustering of soccer players. Proceedings of the International Machine Vision and Image Processing Conference (IMVIP 2007), Washington, DC, USA.10.1109/IMVIP.2007.37 ↗
17. Le Troter, A., Mavromatis, S., and Sequeira, J. (October, January 29). Soccer field detection in video images using color and spatial coherence. Proceedings of the International Conference Image Analysis and Recognition, Porto, Portugal.
18. Heydari, M., and Moghadam, A.M.E. (2012, January 22–23). An MLP-based player detection and tracking in broadcast soccer video. Proceedings of the International Conference of Robotics and Artificial Intelligence, Rawalpindi, Pakistan.10.1109/ICRAI.2012.6413398 ↗
19. Barnard, M., and Odobez, J.M. (2004, January 26). Robust playfield segmentation using MAP adaptation. Proceedings of the 17th International Conference on Pattern Recognition, Cambridge, UK.10.1109/ICPR.2004.1334603 ↗
20. Graph-based multiplayer detection and tracking in broadcast soccer videos10.1109/TMM.2008.922869 ↗
21. Ul Huda, N., Jensen, K.H., Gade, R., and Moeslund, T.B. (2018, January 18–22). Estimating the number of soccer players using simulation-based occlusion handling. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00236 ↗
22. Ohno, Y., Miura, J., and Shirai, Y. (2000, January 3–7). Tracking players and estimation of the 3D position of a ball in soccer games. Proceedings of the 15th International Conference on Pattern Recognition, Barcelona, Spain.
23. Santiago, C.B., Sousa, A., Reis, L.P., and Estriga, M.L. (2011). Real time colour based player tracking in indoor sports. Computational Vision and Medical Image Processing, Springer.10.1007/978-94-007-0011-6_2 ↗
24. Tracking the soccer ball using multiple fixed cameras10.1016/j.cviu.2008.01.007 ↗
25. Kasuya, N., Kitahara, I., Kameda, Y., and Ohta, Y. (2010, January 25–29). Real-time soccer player tracking method by utilizing shadow regions. Proceedings of the 18th ACM International Conference on Multimedia, Firenze, Italy.10.1145/1873951.1874211 ↗
26. Homayounfar, N., Fidler, S., and Urtasun, R. (2017, January 21–26). Sports field localization via deep structured models. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.427 ↗
27. Leo, M., Mosca, N., Spagnolo, P., Mazzeo, P.L., D’Orazio, T., and Distante, A. (2008, January 7–9). Real-time multiview analysis of soccer matches for understanding interactions between ball and players. Proceedings of the 2008 International Conference on Content-Based Image and Video Retrieval, Niagara Falls, ON, Canada.10.1145/1386352.1386419 ↗
28. Conaire, C.O., Kelly, P., Connaghan, D., and O’Connor, N.E. (2009, January 5–7). Tennissense: A platform for extracting semantic information from multi-camera tennis data. Proceedings of the 16th International Conference on Digital Signal Processing, Santorini, Greece.10.1109/ICDSP.2009.5201152 ↗
29. Ontology-based global and collective motion patterns for event classification in basketball videos
30. Fusing motion patterns and key visual information for semantic event recognition in basketball videos10.1016/j.neucom.2020.07.003 ↗
31. Objects detection toward complicated high remote basketball sports by leveraging deep CNN architecture10.1016/j.future.2021.01.020 ↗
32. Multiple player tracking in basketball court videos10.1007/s11554-020-00968-x ↗
33. Analyzing basketball movements and pass relationships using realtime object tracking techniques based on deep learning10.1109/ACCESS.2019.2913953 ↗
34. Ramanathan, V., Huang, J., Abu-El-Haija, S., Gorban, A., Murphy, K., and Fei-Fei, L. (2016, January 27–30). Detecting events and key actors in multi-person videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.332 ↗
35. A real-time trajectory-based ball detection-and-tracking framework for basketball video10.1007/s12596-012-0108-7 ↗
36. An automated player detection and tracking in basketball game
37. Acuna, D. (2017, January 4–9). Towards real-time detection and tracking of basketball players using deep neural networks. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
38. Applying deep bidirectional LSTM and mixture density network for basketball trajectory prediction10.1016/j.ijleo.2017.12.038 ↗
39. Shah, R., and Romijnders, R. (2016). Applying Deep Learning to Basketball Trajectories. arXiv.
40. Recognition of basketball referee signals from real-time videos10.1007/s12652-019-01209-1 ↗
41. Deep learning based basketball video analysis for intelligent arena application10.1007/s11042-017-5002-5 ↗
42. Real-Time Analysis of Basketball Sports Data Based on Deep Learning10.1155/2021/9142697 ↗
43. Analysis of technical features in basketball video based on deep learning algorithm10.1016/j.image.2020.115786 ↗
44. Wang, K.C., and Zemel, R. (2016, January 11–12). Classifying NBA offensive plays using neural networks. Proceedings of the Proceedings of MIT Sloan Sports Analytics Conference, Boston, MA, USA.
45. End-to-End Key-Player-Based Group Activity Recognition Network Applied to Basketball Offensive Tactic Identification in Limited Data Scenarios10.1109/ACCESS.2021.3098840 ↗
46. Space creation dynamics in basketball offence: Validation and evaluation of elite teams10.1080/24748668.2011.11868530 ↗
47. Space–time coordination dynamics in basketball: Part 1. Intra-and inter-couplings among player dyads10.1080/02640410903503632 ↗
48. Space–time coordination dynamics in basketball: Part 2. The interaction between the two teams10.1080/02640410903503640 ↗
49. Tian, C., De Silva, V., Caine, M., and Swanson, S. (2020). Use of machine learning to automate the identification of basketball strategies using whole team player tracking data. Appl. Sci., 10.10.3390/app10010024 ↗
50. Hauri, S., Djuric, N., Radosavljevic, V., and Vucetic, S. (2021, January 3–8). Multi-Modal Trajectory Prediction of NBA Players. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA.10.1109/WACV48630.2021.00168 ↗
51. Zheng, S., Yue, Y., and Lucey, P. (2016, January 5–10). Generating Long-Term Trajectories Using Deep Hierarchical Networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain.
52. AC-VRNN: Attentive Conditional-VRNN for multi-future trajectory prediction10.1016/j.cviu.2021.103245 ↗
53. Enhancing trajectory prediction using sparse outputs: Application to team sports10.1007/s00521-021-05888-w ↗
54. Artificial Intelligence and Neural Network-Based Shooting Accuracy Prediction Analysis in Basketball
55. Recognizing tactic patterns in broadcast basketball video using player trajectory10.1016/j.jvcir.2012.06.003 ↗
56. Physics-based ball tracking and 3D trajectory reconstruction with applications to shooting location estimation in basketball video10.1016/j.jvcir.2008.11.008 ↗
57. Design of basketball game image acquisition and processing system based on machine vision and image processor10.1016/j.micpro.2021.103904 ↗
58. Lineup Optimization Model of Basketball Players Based on the Prediction of Recursive Neural Networks
59. Suda, S., Makino, Y., and Shinoda, H. (2019, January 11–12). Prediction of volleyball trajectory using skeletal motions of setter player. Proceedings of the 10th Augmented Human International Conference, Reims, France.10.1145/3311823.3311844 ↗
60. Soccer player recognition using spatial constellation features and jersey number recognition10.1016/j.cviu.2017.04.010 ↗
61. Sentioscope: A soccer player tracking system using model field particles10.1109/TCSVT.2015.2455713 ↗
62. A deep learning ball tracking system in soccer videos10.1016/j.opelre.2019.02.003 ↗
63. Automatic initialization for 3D soccer player tracking10.1016/j.patrec.2011.03.009 ↗
64. Multiple object tracking in soccer videos using topographic surface analysis10.1016/j.jvcir.2019.102683 ↗
65. Automatic player detection, labeling and tracking in broadcast soccer video10.1016/j.patrec.2008.02.011 ↗
66. Komorowski, J., Kurzejamski, G., and Sarwas, G. (2019, January 27–31). BallTrack: Football ball tracking for real-time CCTV systems. Proceedings of the 16th International Conference on Machine Vision Applications (MVA), Tokyo, Japan.10.23919/MVA.2019.8757880 ↗
67. Hurault, S., Ballester, C., and Haro, G. (2020, January 12–16). Self-Supervised Small Soccer Player Detection and Tracking. Proceedings of the 3rd International Workshop on Multimedia Content Analysis in Sports, Seattle, WA, USA.10.1145/3422844.3423054 ↗
68. Kamble, P.R., Keskar, A.G., and Bhurchandi, K.M. (2018, January 1–3). A convolutional neural network based 3D ball tracking by detection in soccer videos. Proceedings of the Eleventh International Conference on Machine Vision (ICMV 2018), Munich, Germany.10.1117/12.2522844 ↗
69. Naidoo, W.C., and Tapamo, J.R. (2006, January 9–11). Soccer video analysis by ball, player and referee tracking. Proceedings of the 2006 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries, Somerset West, South Africa.10.1145/1216262.1216268 ↗
70. Liang, D., Liu, Y., Huang, Q., and Gao, W. (2005, January 13–16). A scheme for ball detection and tracking in broadcast soccer video. Proceedings of the Pacific-Rim Conference on Multimedia, Jeju Island, Korea.10.1007/11581772_76 ↗
71. YOLOv3-SORT detection and tracking player-ball in soccer sport10.1117/1.JEI.32.1.011003 ↗
72. DeepPlayer-Track: Player and Referee Tracking with Jersey Color Recognition in Soccer10.1109/ACCESS.2022.3161441 ↗
73. Komorowski, J., Kurzejamski, G., and Sarwas, G. (2020, January 27–29). FootAndBall: Integrated Player and Ball Detector. Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Valletta, Malta.10.5220/0008916000470056 ↗
74. Ball detection from broadcast soccer videos using static and dynamic features10.1016/j.jvcir.2008.06.007 ↗
75. Accurate ball detection in soccer images using probabilistic analysis of salient regions10.1007/s00138-013-0518-9 ↗
76. Soccer ball detection by comparing different feature extraction methodologies10.1155/2012/512159 ↗
77. Garnier, P., and Gregoir, T. (2021). Evaluating Soccer Player: From Live Camera to Deep Reinforcement Learning. arXiv.
78. Machine Learning Enabled Team Performance Analysis in the Dynamical Environment of Soccer10.1109/ACCESS.2020.2992025 ↗
79. Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., and Baskurt, A. (2010, January 15–18). Action classification in soccer videos with long short-term memory recurrent neural networks. Proceedings of the International Conference on Artificial Neural Networks, Thessaloniki, Greece.10.1007/978-3-642-15822-3_20 ↗
80. Jackman, S. (2019). Football Shot Detection Using Convolutional Neural Networks. [Master’s Thesis, Department of Biomedical Engineering].
81. Lucey, P., Bialkowski, A., Monfort, M., Carr, P., and Matthews, I. (March, January 28). quality vs quantity: Improved shot prediction in soccer using strategic features from spatiotemporal data. Proceedings of the 8th Annual MIT Sloan Sports Analytics Conference, Boston, MA, USA.
82. Cioppa, A., Deliege, A., Giancola, S., Ghanem, B., Droogenbroeck, M.V., Gade, R., and Moeslund, T.B. (2020, January 13–19). A context-aware loss function for action spotting in soccer videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.01314 ↗
83. Beernaerts, J., De Baets, B., Lenoir, M., and Van de Weghe, N. (2020). Spatial movement pattern recognition in soccer based on relative player movements. PLoS ONE, 15.10.1371/journal.pone.0227746 ↗
84. Sport action mining: Dribbling recognition in soccer10.1007/s11042-021-11784-1 ↗
85. Kim, Y., Jung, C., and Kim, C. (2021, November 02). Motion Recognition of Assistant Referees in Soccer Games via Selective Color Contrast Revelation. EasyChair Preprint no. 2604, EasyChair. Available online: https://easychair.org/publications/preprint/z975.
86. Lindström, P., Jacobsson, L., Carlsson, N., and Lambrix, P. (2020, January 14–18). Predicting player trajectories in shot situations in soccer. Proceedings of the International Workshop on Machine Learning and Data Mining for Sports Analytics, Ghent, Belgium.10.1007/978-3-030-64912-8_6 ↗
87. Visual soccer match analysis using spatiotemporal positions of players10.1016/j.cag.2017.08.006 ↗
88. Ganesh, Y., Teja, A.S., Munnangi, S.K., and Murthy, G.R. (2019, January 17–19). A Novel Framework for Fine Grained Action Recognition in Soccer. Proceedings of the International Work-Conference on Artificial Neural Networks, Munich, Germany.10.1007/978-3-030-20518-8_12 ↗
89. Classification of passes in football matches using spatiotemporal data10.1145/3105576 ↗
90. Gyarmati, L., and Stanojevic, R. (2016). QPass: A Merit-based Evaluation of Soccer Passes. arXiv.
91. Vercruyssen, V., De Raedt, L., and Davis, J. (2016). Qualitative spatial reasoning for soccer pass prediction. CEUR Workshop Proceedings, Springer.
92. Yu, J., Lei, A., and Hu, Y. (2019, January 8–11). Soccer video event detection based on deep learning. Proceedings of the International Conference on Multimedia Modeling, Thessaloniki, Greece.10.1007/978-3-030-05716-9_31 ↗
93. Using machine learning to draw inferences from pass location data in soccer10.1002/sam.11318 ↗
94. Pass2vec: Analyzing soccer players’ passing style using deep learning10.1177/17479541211033078 ↗
95. An automatic multi-camera-based event extraction system for real soccer videos10.1007/s10044-019-00830-2 ↗
96. Deliège, A., Cioppa, A., Giancola, S., Seikavandi, M.J., Dueholm, J.V., Nasrollahi, K., Ghanem, B., Moeslund, T.B., and Droogenbroeck, M.V. (2020). SoccerNet-v2: A Dataset and Benchmarks for Holistic Understanding of Broadcast Soccer Videos. arXiv.10.1109/CVPRW53098.2021.00508 ↗
97. Automated Goal Score Detection in Football Match Using Key Moments10.1016/j.procs.2020.01.017 ↗
98. Khan, A., Lazzerini, B., Calabrese, G., and Serafini, L. (2018, January 28–29). Soccer event detection. Proceedings of the 4th International Conference on Image Processing and Pattern Recognition (IPPR 2018), Copenhagen, Denmark.10.5121/csit.2018.80509 ↗
99. Khaustov, V., and Mozgovoy, M. (2020). Recognizing Events in Spatiotemporal Soccer Data. Appl. Sci., 10.10.3390/app10228046 ↗
100. Saraogi, H., Sharma, R.A., and Kumar, V. (2016, January 18–22). Event recognition in broadcast soccer videos. Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, Hyderabad, India.10.1145/3009977.3010074 ↗