SelectStitch: Automated Frame Segmentation and Stitching to Create Composite Images from Otoscope Video Clips
Source abstract
Abstract
Background and Objective: the aim of this study is to develop and validate an automated image segmentation-based frame selection and stitching framework to create enhanced composite images from otoscope videos. The proposed framework, called SelectStitch, is useful for classifying eardrum abnormalities using a single composite image instead of the entire raw otoscope video dataset. Methods: SelectStitch consists of a convolutional neural network (CNN) based semantic segmentation approach to detect the eardrum in each frame of the otoscope video, and a stitching engine to generate a high-quality composite image from the detected eardrum regions. In this study, we utilize two separate datasets: the first one has 36 otoscope videos that were used to train a semantic segmentation model, and the second one, containing 100 videos, which was used to test the proposed method. Cases from both adult and pediatric patients were used in this study. A configuration of 4-levels depth U-Net architecture was trained to automatically find eardrum regions in each otoscope video frame from the first dataset. After the segmentation, we automatically selected meaningful frames from otoscope videos by using a pre-defined threshold, i.e., it should contain at least an eardrum region of 20% of a frame size. We have generated 100 composite images from the test dataset. Three ear, nose, and throat (ENT) specialists (ENT-I, ENT-II, ENT-III) compared in two rounds the composite images produced by SelectStitch against the composite images that were generated by the base processes, i.e., stitching all the frames from the same video data, in terms of their diagnostic capabilities. Results: In the first round of the study, ENT-I, ENT-II, ENT-III graded improvement for 58, 57, and 71 composite images out of 100, respectively, for SelectStitch over the base composite, reflecting greater diagnostic capabilities. In the repeat assessment, these numbers were 56, 56, and 64, respectively. We observed that only 6%, 3%, and 3% of the cases received a lesser score than the base composite images, respectively, for ENT-I, ENT-II, and ENT-III in Round-1, and 4%, 0%, and 2% of the cases in Round-2. Conclusions: We conclude that the frame selection and stitching will increase the probability of detecting a lesion even if it appears in a few frames.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Reconstructive Facial Surgery Techniques · Ear Surgery and Otitis Media · Facial Nerve Paralysis Treatment and Research
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Biomedical Informatics, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
Department of Otolaryngology, Ohio State University, Columbus, OH 43212, USA
Case Western Reserve University School of Medicine, Cleveland, OH 44106, USA
University Hospitals Seidman Cancer Center, Cleveland, OH 44106, USA
Department of Internal Medicine, Ohio State University College of Medicine, Columbus, OH 43210, USA
Department of Biomedical Informatics, Ohio State University, Columbus, OH 43210, USA
Funding
National Institute on Deafness and Other Communication Disorders
References (33)
Show source-provided references
1. Decision fusion on image analysis and tympanometry to detect eardrum abnormalities
2. A Mosaicking Approach for In Vivo Thickness Mapping of the Human Tympanic Membrane Using Low Coherence Interferometry10.1007/s10162-016-0576-6 ↗
3. Noninvasive in vivo optical coherence tomography tracking of chronic otitis media in pediatric subjects after surgical intervention10.1117/1.JBO.22.12.121614 ↗
4. Video pneumatic otoscopy for the diagnosis of otitis media with effusion: A quantitative approach10.1007/s00405-008-0839-x ↗
5. Automated Diagnosis of Otitis Media: Vocabulary and Grammar10.1155/2013/327515 ↗
6. Smartphone-Enabled Otoscopy in Neurotology/Otology10.1177/0194599816687740 ↗
7. Assessment of a Smartphone Otoscope Device for the Diagnosis and Management of Otitis Media10.1177/0009922815593909 ↗
8. Digital otoscopy versus microscopy: How correct and confident are ear experts in their diagnoses?10.1177/1357633X17708531 ↗
9. Otitis Media Diagnosis for Developing Countries Using Tympanic Membrane Image-Analysis10.1016/j.ebiom.2016.02.017 ↗
10. Detection of eardrum abnormalities using ensemble deep learning approaches
11. Autoscope: Automated otoscopy image analysis to diagnose ear pathology and use of clinically motivated eardrum features
12. Digital pathology and artificial intelligence10.1016/S1470-2045(19)30154-8 ↗
13. Kasher, M.S. (2018). Otitis Media Analysis-An Automated Feature Extraction and Image Classification System. [Bachelor’s Thesis, Helsinki Metropolia University of Applied Sciences].
14. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-24574-4_28 ↗
15. Long, J., Shelhamer, E., and Darrell, T. (2015, January 7–12). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298965 ↗
16. Noh, H., Hong, S., and Han, B. (2015, January 7–13). Learning deconvolution network for semantic segmentation. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.178 ↗
17. Big Data Deep Learning: Challenges and Perspectives10.1109/ACCESS.2014.2325029 ↗
18. Simard, P.Y., Steinkraus, D., and Platt, J.C. (2003, January 6). Best practices for convolutional neural networks applied to visual document analysis. Proceedings of the Icdar, Edinburgh, UK.
19. Jung, A.B. (2019, January 10). Imgaug. Available online: https://github.com/aleju/imgaug.
20. McLachlan, G.J., Do, K.-A., and Ambroise, C. (2005). Analyzing Microarray Gene Expression Data, John Wiley & Sons.10.1002/047172842X ↗
21. Bay, H., Tuytelaars, T., and Van Gool, L. (2006). Surf: Speeded up robust features. European Conference on Computer Vision, Springer.10.1007/11744023_32 ↗
22. Microsoft (2018, December 20). Image Composite Editor (ICE). Available online: https://www.microsoft.com/en-us/research/product/computational-photography-applications/image-composite-editor/.
23. Jain, A., and Gupta, R. (2015). Gaussian filter threshold modulation for filtering flat and texture area of an image. 2015 International Conference on Advances in Computer Engineering and Applications, IEEE.10.1109/ICACEA.2015.7164804 ↗
24. A Threshold Selection Method from Gray-Level Histograms10.1109/TSMC.1979.4310076 ↗
25. Dong, X., Wang, G., Pang, Y., Li, W., Wen, J., Meng, W., and Lu, Y. (2011). Fast efficient algorithm for enhancement of low lighting video. 2011 IEEE International Conference on Multimedia and Expo, IEEE.10.1109/ICME.2011.6012107 ↗
26. Measures of the Amount of Ecologic Association between Species10.2307/1932409 ↗
27. A multidimensional scaling and sample clustering to obtain a representative subset of training data for transfer learning-based rosacea lesion identification
28. Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.
29. Goodfellow, I., Bengio, Y., and Courville, A. (2016). Deep Learning, MIT Press.
30. Ros-NET: A deep convolutional neural network for automatic identification of rosacea lesions10.1111/srt.12817 ↗
31. Assessing observer variability: A user’s guide10.21037/cdt.2017.03.12 ↗
32. Morphometric analysis of white matter lesions in MR images: Method and validation10.1109/42.363096 ↗
33. Lee, J.Y., Chung, J.W., and Chung, J.W. (2019). Automated Classification of the Tympanic Membrane Using a Convolutional Neural Network. Appl. Sci., 9.10.3390/app9091827 ↗