Surgical Tool Detection in Open Surgery Videos
Source abstract
Abstract
Detecting surgical tools is an essential task for analyzing and evaluating surgical videos. However, most studies focus on minimally invasive surgery (MIS) and cataract surgery. Mainly because of a lack of a large, diverse, and well-annotated dataset, research in the area of open surgery has been limited so far. Open surgery video analysis is challenging because of its properties: varied number and roles of people (e.g., main surgeon, assistant surgeons, and nurses), a complex interaction of tools and hands, various operative environments, and lighting conditions. In this paper, to handle these limitations and difficulties, we introduce an egocentric open surgery dataset that includes 15 open surgeries recorded with a head-mounted camera. More than 67k bounding boxes are labeled to 19k images with 31 surgical tool categories. Finally, we present a surgical tool detection baseline model based on recent advances in object detection. The results of our new dataset show that our presented dataset provides enough interesting challenges for future methods and that it can serve as a strong benchmark to address the study of tool detection in open surgery.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Surgical Simulation and Training · Digital Imaging in Medicine · Intraocular Surgery and Lenses
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Science and Technology, Keio University, Yokohama 223-8852, Japan
Department of Plastic and Reconstructive Surgery, Keio University School of Medicine, Tokyo 160-8582, Japan
Funding
MHLW Health, Labour, and Welfare Sciences Research Grants Research on Medical ICT and Artificial Intelligence Program · MIC/SCOPE · JSPS KAKENHI
References (50)
Show source-provided references
1. Schoeffmann, K., Chalidabhongse, T.H., Ngo, C.W., Aramvith, S., O’Connor, N.E., Ho, Y.S., Gabbouj, M., and Elgammal, A. (2018). Frame-Based Classification of Operation Phases in Cataract Surgery Videos. MultiMedia Modeling, Springer International Publishing.10.1007/978-3-319-73600-6 ↗
2. Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., and Fichtinger, G. (2018). DeepPhase: Surgical Phase Recognition in CATARACTS Videos. Medical Image Computing and Computer Assisted Intervention, Springer International Publishing.10.1007/978-3-030-00934-2 ↗
3. Future-State Predicting LSTM for Early Surgery Type Recognition10.1109/TMI.2019.2931158 ↗
4. Volkov, M., Hashimoto, D.A., Rosman, G., Meireles, O.R., and Rus, D. (June, January 29). Machine learning and coresets for automated real-time video segmentation of laparoscopic and robot-assisted surgery. Proceedings of the International Conference on Robotics and Automation (ICRA), Singapore.
5. Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., and Joskowicz, L. (2020). Ultrasound Video Summarization Using Deep Reinforcement Learning. Medical Image Computing and Computer Assisted Intervention, Springer International Publishing.10.1007/978-3-030-59716-0 ↗
6. Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos10.1007/s11548-019-01958-6 ↗
7. Combined 2D and 3D tracking of surgical instruments for minimally invasive and robotic-assisted surgery10.1007/s11548-016-1393-4 ↗
8. Chen, Z., Zhao, Z., and Cheng, X. (2017, January 20–22). Surgical instruments tracking based on deep learning with lines detection and spatio-temporal context. Proceedings of the Chinese Automation Congress (CAC), Jinan, China.10.1109/CAC.2017.8243236 ↗
9. Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., and Duchesne, S. (2017). Simultaneous Recognition and Pose Estimation of Instruments in Minimally Invasive Surgery. Medical Image Computing and Computer-Assisted Intervention, Springer International Publishing.10.1007/978-3-319-66185-8 ↗
10. Deep Learning Based Robotic Tool Detection and Articulation Estimation With Spatio-Temporal Layers10.1109/LRA.2019.2917163 ↗
11. Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks10.1109/TMI.2017.2787672 ↗
12. Zhou, L., Sarikaya, D., Kia, S.M., Speidel, S., Malpani, A., Hashimoto, D., Habes, M., Löfstedt, T., Ritter, K., and Wang, H. (2019). Unsupervised Temporal Video Segmentation as an Auxiliary Task for Predicting the Remaining Surgery Duration. OR 2.0 Context-Aware Operating Theaters and Machine Learning in Clinical Neuroimaging, Springer International Publishing.10.1007/978-3-030-32695-1 ↗
13. Jin, A., Yeung, S., Jopling, J., Krause, J., Azagury, D., Milstein, A., and Fei-Fei, L. (2018, January 4–8). Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA.10.1109/WACV.2018.00081 ↗
14. Choi, B., Jo, K., Choi, S., and Choi, J. (2017, January 11–15). Surgical-tools detection based on Convolutional Neural Network in laparoscopic robot-assisted surgery. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju, Korea.10.1109/EMBC.2017.8037183 ↗
15. Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection10.1109/TMI.2017.2665671 ↗
16. Vardazaryan, A., Mutter, D., Marescaux, J., and Padoy, N. (2018). Weakly-supervised learning for tool localization in laparoscopic videos. MICCAI LABELS, Springer.10.1007/978-3-030-01364-6_19 ↗
17. CATARACTS: Challenge on automatic tool annotation for cataRACT surgery10.1016/j.media.2018.11.008 ↗
18. EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos10.1109/TMI.2016.2593957 ↗
19. Twinanda, A.P., Mutter, D., Marescaux, J., de Mathelin, M., and Padoy, N. (2016). Single-and multi-task architectures for tool presence detection challenge at M2CAI 2016. arXiv.
20. Bawa, V.S., Singh, G., KapingA, F., Skarga-Bandurova, I., Oleari, E., Leporini, A., Landolfo, C., Zhao, P., Xiang, X., and Luo, G. (2021). The saras endoscopic surgeon action detection (esad) dataset: Challenges and methods. arXiv.
21. Navab, N., Hornegger, J., Wells, W.M., and Frangi, A. (2015). Automated Assessment of Surgical Skills Using Frequency Analysis. Medical Image Computing and Computer-Assisted Intervention, Springer International Publishing.10.1007/978-3-319-24571-3 ↗
22. Basiev, K., Goldbraikh, A., Pugh, C.M., and Laufer, S. (2021). Open surgery tool classification and hand utilization using a multi-camera system. arXiv.10.1007/s11548-022-02691-3 ↗
23. Video-based fully automatic assessment of open surgery suturing skills10.1007/s11548-022-02559-6 ↗
24. Shimizu, T., Hachiuma, R., Kajita, H., Takatsume, Y., and Saito, H. (2021). Hand Motion-Aware Surgical Tool Localization and Classification from an Egocentric Camera. J. Imaging, 7.10.3390/jimaging7020015 ↗
25. Goodman, E.D., Patel, K.K., Zhang, Y., Locke, W., Kennedy, C.J., Mehrotra, R., Ren, S., Guan, M., Downing, M., and Chen, H.W. (2021). A real-time spatiotemporal AI model analyzes skill in open surgical videos. arXiv.
26. Farinella, G., Radeva, P., and Braz, J. (2020). Surgery recording without occlusions by multi-view surgical videos. VISAPP, Proceedings of the International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP), Valetta, Malta, 27–29 February 2020, SciTePress.
27. Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (2015). Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. Advances in Neural Information Processing Systems (NIPS), Curran Associates Inc.
28. Focal Loss for Dense Object Detection10.1109/TPAMI.2018.2858826 ↗
29. Real-time instrument detection in minimally invasive surgery using radiofrequency identification technology10.1016/j.jss.2013.06.022 ↗
30. Haase, S., Wasza, J., Kilgus, T., and Hornegger, J. (2013, January 15–17). Laparoscopic instrument localization using a 3-D Time-of-Flight/RGB endoscope. Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV), Clearwater Beach, FL, USA.10.1109/WACV.2013.6475053 ↗
31. Reiter, A., and Allen, P.K. (2010, January 18–22). An online learning approach to in-vivo tracking using synergistic features. Proceedings of the 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei, Taiwan.10.1109/IROS.2010.5650852 ↗
32. Detecting Surgical Tools by Modelling Local Appearance and Global Shape10.1109/TMI.2015.2450831 ↗
33. Ayache, N., Delingette, H., Golland, P., and Mori, K. (2012). Feature Classification for Tracking Articulated Surgical Tools. Medical Image Computing and Computer-Assisted Intervention, Springer.
34. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You Only Look Once: Unified, Real-Time Object Detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
35. Mishra, K., Sathish, R., and Sheet, D. (2017, January 21–26). Learning Latent Temporal Connectionism of Deep Residual Visual Abstractions for Identifying Surgical Tools in Laparoscopy Procedures. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Honolulu, HI, USA.10.1109/CVPRW.2017.277 ↗
36. Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks10.1016/j.media.2018.05.001 ↗
37. Chung, A.C.S., Gee, J.C., Yushkevich, P.A., and Bao, S. (2019). Graph Convolutional Nets for Tool Presence Detection in Surgical Videos. Information Processing in Medical Imaging, Springer International Publishing.10.1007/978-3-030-20351-1 ↗
38. Multi-task recurrent convolutional network with correlation loss for surgical video analysis10.1016/j.media.2019.101572 ↗
39. Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., and Joskowicz, L. (2020). Deep Selection: A Fully Supervised Camera Selection Network for Surgery Recordings. Medical Image Computing and Computer Assisted Intervention, Springer International Publishing.10.1007/978-3-030-59713-9 ↗
40. Camera Selection for Occlusion-Less Surgery Recording via Training With an Egocentric Camera10.1109/ACCESS.2021.3118426 ↗
41. Spatiotemporal Video Highlight by Neural Network Considering Gaze and Hands of Surgeon in Egocentric Surgical Videos10.1142/S2424905X21410014 ↗
42. Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery
43. Real-Time Surgical Tool Detection in Minimally Invasive Surgery Based on Attention-Guided Convolutional Neural Network10.1109/ACCESS.2020.3046258 ↗
44. A Dataset and Benchmarks for Segmentation and Recognition of Gestures in Robotic Surgery10.1109/TBME.2016.2647680 ↗
45. Jhu-Isi Gesture and Skill Assessment Working Set (Jigsaws): A Surgical Activity Dataset for Human Motion Modeling
46. Kadkhodamohammadi, A., Gangi, A., de Mathelin, M., and Padoy, N. (2017, January 24–31). A Multi-view RGB-D Approach for Human Pose Estimation in Operating Rooms. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA.10.1109/WACV.2017.47 ↗
47. He, K., Zhang, X., Ren, S., and Sun, J. (July, January 26). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.
48. Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K. (2017, January 22–25). Aggregated Residual Transformations for Deep Neural Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.634 ↗
49. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., and Girshick, R. (2022, August 31). Detectron2. Available online: https://github.com/facebookresearch/detectron2.
50. Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (2014). Microsoft COCO: Common Objects in Context. Computer Vision—ECCV 2014, Springer International Publishing.10.1007/978-3-319-10590-1 ↗