Development of a Deep Learning-Based Algorithm to Detect the Distal End of a Surgical Instrument
Source abstract
Abstract
This work aims to develop an algorithm to detect the distal end of a surgical instrument using object detection with deep learning. We employed nine video recordings of carotid endarterectomies for training and testing. We obtained regions of interest (ROI; 32 × 32 pixels), at the end of the surgical instrument on the video images, as supervised data. We applied data augmentation to these ROIs. We employed a You Only Look Once Version 2 (YOLOv2) -based convolutional neural network as the network model for training. The detectors were validated to evaluate average detection precision. The proposed algorithm used the central coordinates of the bounding boxes predicted by YOLOv2. Using the test data, we calculated the detection rate. The average precision (AP) for the ROIs, without data augmentation, was 0.4272 ± 0.108. The AP with data augmentation, of 0.7718 ± 0.0824, was significantly higher than that without data augmentation. The detection rates, including the calculated coordinates of the center points in the centers of 8 × 8 pixels and 16 × 16 pixels, were 0.6100 ± 0.1014 and 0.9653 ± 0.0177, respectively. We expect that the proposed algorithm will be efficient for the analysis of surgical records.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Medical Imaging and Analysis · Surgical Simulation and Training · Medical Image Segmentation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Faculty of Health Sciences, Hokkaido University, Sapporo 060-0812, Japan
Department of Neurosurgery, Hokkaido University Graduate School of Medicine, Sapporo 060-8638, Japan
Graduate School of Health Sciences, Hokkaido University, Sapporo 060-0812, Japan
References (45)
Show source-provided references
1. Classification of computed tomography images in different slice positions using deep learning10.1155/2018/1753480 ↗
2. Sugimori, H. (2019). Evaluating the overall accuracy of additional learning and automatic classification system for CT images. Appl. Sci., 9.10.3390/app9040682 ↗
3. Brain tumor classification for MR images using transfer learning and fine-tuning10.1016/j.compmedimag.2019.05.001 ↗
4. Liver fibrosis: Deep convolutional neural network for staging by using gadoxetic acid–enhanced hepatobiliary phase MR images10.1148/radiol.2017171928 ↗
5. MRI Gibbs-ringing artifact reduction by means of machine learning using convolutional neural networks10.1002/mrm.27894 ↗
6. Roth, H.R., Lee, C.T., Shin, H.-C., Seff, A., Kim, L., Yao, J., Lu, L., and Summers, R.M. (2015, January 16–19). Anatomy-specific classification of medical images using deep convolutional nets. Proceedings of the 2015 IEEE International Symposium on Biomedical Imaging, New York, NY, USA.10.1109/ISBI.2015.7163826 ↗
7. Artificial intelligence using neural network architecture for radiology (AINNAR): Classification of MR imaging sequences10.1007/s11604-018-0779-3 ↗
8. Classification of CT brain images based on deep learning networks10.1016/j.cmpb.2016.10.007 ↗
9. Sugimori, H., and Kawakami, M. (2019). Automatic detection of a standard line for brain magnetic resonance imaging using deep learning. Appl. Sci., 9.10.3390/app9183849 ↗
10. An experimental study on breast lesion detection and classification from ultrasound images using deep learning architectures10.1186/s12880-019-0349-x ↗
11. Automatic detection and classification of radiolucent lesions in the mandible on panoramic radiographs using a deep learning object detection technique10.1016/j.oooo.2019.05.014 ↗
12. Ge, C., Gu, I.Y.H., Jakola, A.S., and Yang, J. (2018, January 17–22). Deep learning and multi-sensor fusion for glioma classification using multistream 2D convolutional networks. Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Honolulu, HI, USA.10.1109/EMBC.2018.8513556 ↗
13. Using deep learning to segment breast and fibroglandular tissue in MRI volumes10.1002/mp.12079 ↗
14. Convolutional neural network for automated FLAIR lesion segmentation on clinical brain MR imaging10.3174/ajnr.A6138 ↗
15. Brain tumor segmentation with deep neural networks10.1016/j.media.2016.05.004 ↗
16. Fully automatic acute ischemic lesion segmentation in DWI using convolutional neural networks10.1016/j.nicl.2017.06.016 ↗
17. Automatic brain tissue segmentation in fetal MRI using convolutional neural networks10.1016/j.mri.2019.05.020 ↗
18. Forces of tool-tissue interaction to assess surgical skill level10.1001/jamasurg.2017.4516 ↗
19. Surgical skill and complication rates after bariatric surgery10.1056/NEJMsa1300625 ↗
20. Robot-assisted minimally invasive surgical skill assessment—Manual and automated platforms
21. Jo, K., Choi, Y., Choi, J., and Chung, J.W. (2019). Robust real-time detection of laparoscopic instruments in robot surgery using convolutional neural networks with motion vector prediction. Appl. Sci., 9.10.3390/app9142865 ↗
22. Real-time surgical instrument detection in robot-assisted surgery using a convolutional neural network cascade10.1049/htl.2019.0064 ↗
23. Vision-based and marker-less surgical tool detection and tracking: A review of the literature10.1016/j.media.2016.09.003 ↗
24. Detecting surgical tools by modelling local appearance and global shape10.1109/TMI.2015.2450831 ↗
25. A pilot study on measuring tissue motion during carotid surgery using video-based analyses for the objective assessment of surgical performance10.1007/s00268-019-05018-7 ↗
26. Real-time tracking of surgical instruments based on spatio-temporal context and deep learning10.1080/24699322.2018.1560097 ↗
27. Machine learning algorithm validation with a limited sample size10.1371/journal.pone.0224365 ↗
28. Mikołajczyk, A., and Grochowski, M. (2018, January 9–12). Data augmentation for improving deep learning in image classification problem. Proceedings of the 2018 International Interdisciplinary Ph.D. Workshop (IIPhDW), Świnoujście, Poland.10.1109/IIPHDW.2018.8388338 ↗
29. Mash, R., Borghetti, B., and Pecarina, J. Improved aircraft recognition for aerial refueling through data augmentation in convolutional neural networks. Proceedings of the Advances in Visual Computing.
30. Taylor, L., and Nitschke, G. (2017). Improving deep learning using generic data augmentation. arXiv.10.1109/SSCI.2018.8628742 ↗
31. Eaton-rosen, Z., and Bragman, F. (2018, January 4–6). Improving data augmentation for medical image segmentation. Proceedings of the 1st Conference on Medical Imaging with Deep Learning (MIDL 2018), Amsterdam, The Netherlands.
32. Redmon, J., and Farhadi, A. (2017, January 21–26). YOLO9000: Better, faster, stronger. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.690 ↗
33. Lalonde, R., Zhang, D., and Shah, M. (2018, January 18–22). ClusterNet: Detecting small objects in large scenes by exploiting spatio-temporal information. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2018, Salt Lake City, UT, USA.10.1109/CVPR.2018.00421 ↗
34. Chen, K., Wang, J., Yang, S., Zhang, X., Xiong, Y., Loy, C.C., and Lin, D. (2018, January 18–22). Optimizing video object detection via a scale-time lattice. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2018, Salt Lake City, UT, USA.10.1109/CVPR.2018.00815 ↗
35. Hasan, S.M.K., and Linte, C.A. (2019, January 23–27). U-NetPlus: A modified encoder-decoder u-net architecture for semantic and instance segmentation of surgical instruments from laparoscopic images. Proceedings of the 41th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany.
36. Ni, Z.-L., Bian, G.-B., Xie, X.-L., Hou, Z.-G., Zhou, X.-H., and Zhou, Y.-J. (2019, January 23–27). RASNet: Segmentation for tracking surgical instruments in surgical videos using refined attention segmentation network. Proceedings of the 41th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Berlin, Germany.10.1109/EMBC.2019.8856495 ↗
37. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 24–27). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2014, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
38. Faster R-CNN: Towards real-time object detection with region proposal networks10.1109/TPAMI.2016.2577031 ↗
39. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016). SSD: Single shot multibox detector. European Conference on Computer Vision, Springer.10.1007/978-3-319-46448-0_2 ↗
40. Preoperative cervical carotid artery contrast-enhanced ultrasound findings are associated with development of microembolic signals on transcranial Doppler during carotid exposure in endarterectomy10.1016/j.atherosclerosis.2017.03.026 ↗
41. Cao, Z., Simon, T., Wei, S.E., and Sheikh, Y. (2017, January 21–26). Realtime multi-person 2D pose estimation using part affinity fields. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.143 ↗
42. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning10.1038/s41593-018-0209-y ↗
43. Papandreou, G., Zhu, T., Chen, L.C., Gidaris, S., Tompson, J., and Murphy, K. (2016). Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. European Conference on Computer Vision, Springer.
44. Allan, M., Shvets, A., Kurmann, T., Zhang, Z., Duggal, R., Su, Y.-H., Rieke, N., Laina, I., Kalavakonda, N., and Bodenstedt, S. (2019). 2017 robotic instrument segmentation challenge. arXiv.
45. Comparative validation of polyp detection methods in video colonoscopy: Results from the MICCAI 2015 endoscopic vision challenge10.1109/TMI.2017.2664042 ↗