journal article·2021-01-12Open access

Accelerating Detection of Lung Pathologies with Explainable Ultrasound Image Analysis

Jannis Born, Nina Wiedemann, Manuel Cossio, Charlotte Buhre, Gabriel Brändle, Konstantin Leidermann, Julie Goulet, Avinash Aujayeb, Michael Moor, Bastian Rieck, Karsten Borgwardt

Applied Sciences· Volume 11· Issue 2

Abstract

Care during the COVID-19 pandemic hinges upon the existence of fast, safe, and highly sensitive diagnostic tools. Considering significant practical advantages of lung ultrasound (LUS) over other imaging techniques, but difficulties for doctors in pattern recognition, we aim to leverage machine learning toward guiding diagnosis from LUS. We release the largest publicly available LUS dataset for COVID-19 consisting of 202 videos from four classes (COVID-19, bacterial pneumonia, non-COVID-19 viral pneumonia and healthy controls). On this dataset, we perform an in-depth study of the value of deep learning methods for the differential diagnosis of lung pathologies. We propose a frame-based model that correctly distinguishes COVID-19 LUS videos from healthy and bacterial pneumonia data with a sensitivity of 0.90±0.08 and a specificity of 0.96±0.04. To investigate the utility of the proposed method, we employ interpretability methods for the spatio-temporal localization of pulmonary biomarkers, which are deemed useful for human-in-the-loop scenarios in a blinded study with medical experts. Aiming for robustness, we perform uncertainty estimation and demonstrate the model to recognize low-confidence situations which also improves performance. Lastly, we validated our model on an independent test dataset and report promising performance (sensitivity 0.806, specificity 0.962). The provided dataset facilitates the validation of related methodology in the community and the proposed framework might aid the development of a fast, accessible screening method for pulmonary diseases. Dataset and all code are publicly available at: https://github.com/BorgwardtLab/covid19_ultrasound.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Ultrasound in Clinical Applications · COVID-19 diagnosis using AI · Phonocardiography and Auscultation Techniques

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Department of Biosystems Science and Engineering, ETH Zurich, 4058 Basel, Switzerland

Department of Computer Science, ETH Zurich, 8092 Zurich, Switzerland

Department of Mathematics and Computer Science, University of Barcelona, 08007 Barcelona, Spain

Brandenburg Medical School Theodor Fontane, 16816 Neuruppin, Germany

Pediatric Emergency Department, Hirslanden Clinique des Grangettes, 1224 Geneva, Switzerland

Department of Philosophy, University of Vienna, 1010 Vienna, Austria

Physik Department T35 and Bernstein Center for Computational Neuroscience, Technische Universität München, 85747 Garching bei München, Germany

Northumbria Specialist Emergency Care Hospital, Cramlington NE23 6NZ, UK

SIB Swiss Institute of Bioinformatics, 1015 Lausanne, Switzerland

References (98)

Show source-provided references
1. Early transmission dynamics in Wuhan, China, of novel coronavirus–infected pneumonia10.1056/NEJMoa2001316 ↗
2. Artificial intelligence–enabled rapid diagnosis of patients with COVID-1910.1038/s41591-020-0931-3 ↗
3. Essentials for radiologists on COVID-19: An update—Radiology scientific expert panel10.1148/radiol.2020200527 ↗
4. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: A report of 1014 cases10.1148/radiol.2020200642 ↗
5. Variation in False-Negative Rate of Reverse Transcriptase Polymerase Chain Reaction–Based SARS-CoV-2 Tests by Time Since Exposure10.7326/M20-1495 ↗
6. Dong, D., Tang, Z., Wang, S., Hui, H., Gong, L., Lu, Y., Xue, Z., Liao, H., Chen, F., and Yang, F. (2020). The role of imaging in the detection and management of COVID-19: A review. IEEE Rev. Biomed. Eng.10.1109/RBME.2020.2990959 ↗
7. Performance comparison of lung ultrasound and chest x-ray for the diagnosis of pneumonia in the ED10.1016/j.ajem.2013.10.003 ↗
8. COVID-19 Computed Tomography Findings: A Systematic Review and Meta-Analysis10.1016/j.jacr.2020.03.006 ↗
9. Sensitivity of chest CT for COVID-19: Comparison to RT-PCR10.1148/radiol.2020200432 ↗
10. Lung ultrasonography versus chest CT in COVID-19 pneumonia: A two-centered retrospective comparison study from China10.1007/s00134-020-06096-1 ↗
11. Radiology department preparedness for COVID-19: Radiology scientific expert panel10.1148/radiol.2020200988 ↗
12. The industry of CT scanning10.3174/ajnr.A2742 ↗
13. Chest x-ray findings in 636 ambulatory patients with COVID-19 presenting to an urgent care center: A normal chest x-ray is no guarantee
14. Use of ultrasound in the developing world10.1186/1865-1380-4-72 ↗
15. Comparative diagnostic performances of auscultation, chest radiography, and lung ultrasonography in acute respiratory distress syndrome10.1097/00000542-200401000-00006 ↗
16. Lung ultrasound for the diagnosis of pneumonia in adults: A systematic review and meta-analysis10.1186/1465-9921-15-50 ↗
17. Lung ultrasound for diagnosis of pneumonia in emergency department10.1007/s11739-015-1297-2 ↗
18. Can lung ultrasound replace chest radiography for the diagnosis of pneumonia in hospitalized children?10.1159/000362692 ↗
19. Performance of chest ultrasound in pediatric pneumonia10.1016/j.ejrad.2016.12.032 ↗
20. Lung ultrasound versus chest radiography for the diagnosis of pneumothorax in critically ill patients: A prospective, single-blind study10.4103/1658-354X.174906 ↗
21. Could the use of bedside lung ultrasound reduce the number of chest X-rays in the intensive care unit?10.1186/s12947-017-0113-8 ↗
22. COVID-19 outbreak: Less stethoscope, more ultrasound10.1016/S2213-2600(20)30120-X ↗
23. Point-of-care lung ultrasound in patients with COVID-19—A narrative review10.1111/anae.15082 ↗
24. The emerging role of lung ultrasound in COVID-19 pneumonia10.5152/eurjrheum.2020.2063 ↗
25. A review of early experience in lung ultrasound (LUS) in the diagnosis and management of COVID-1910.1016/j.ultrasmedbio.2020.05.012 ↗
26. Guiding ultrasound image capture with artificial intelligence10.1016/S0140-6736(20)31875-4 ↗
27. What’s new in lung ultrasound during the COVID-19 pandemic10.1007/s00134-020-06048-9 ↗
28. Point-of-care lung ultrasound is more sensitive than chest radiograph for evaluation of COVID-1910.5811/westjem.2020.5.47743 ↗
29. Findings of lung ultrasonography of novel corona virus pneumonia during the 2019–2020 epidemic10.1007/s00134-020-05996-6 ↗
30. Fiala, M. (2020). Ultrasound in COVID-19: A timeline of ultrasound findings in relation to CT. Clin. Radiol.10.1016/j.crad.2020.04.003 ↗
31. Lieveld, A., Kok, B., Schuit, F., Azijli, K., Heijmans, J., van Laarhoven, A., Assman, N., Kootte, R., Olgers, T., and Nanayakkara, P. (2020). Diagnosing COVID-19 pneumonia in a pandemic setting: Lung Ultrasound versus CT (LUVCT) A multi-centre, prospective, observational study. ERJ Open Res.10.1183/23120541.00539-2020 ↗
32. Correlation between Chest Computed Tomography and Lung Ultrasonography in Patients with Coronavirus Disease 2019 (COVID-19)10.1016/j.ultrasmedbio.2020.07.003 ↗
33. Lung ultrasound as a diagnostic tool for radiographically-confirmed pneumonia in low resource settings10.1016/j.rmed.2017.05.007 ↗
34. Diagnostic use of lung ultrasound compared to chest radiograph for suspected pneumonia in a resource-limited setting10.1186/s12245-018-0170-2 ↗
35. Trends in Ultrasound Use in Low and Middle Income Countries: A Systematic Review
36. The lung ultrasound: Facts or artifacts? In the era of COVID-19 outbreak10.1007/s11547-020-01236-5 ↗
37. Time needed to achieve completeness and accuracy in bedside lung ultrasound reporting in intensive care unit10.1186/1757-7241-18-44 ↗
38. COVID-19 Control by Computer Vision Approaches: A Survey10.1109/ACCESS.2020.3027685 ↗
39. Shi, F., Wang, J., Shi, J., Wu, Z., Wang, Q., Tang, Z., He, K., Shi, Y., and Shen, D. (2020). Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for COVID-19. IEEE Rev. Biomed. Eng.10.1109/RBME.2020.2987975 ↗
40. Born, J., Beymer, D., Rajan, D., Coy, A., Mukherjee, V.V., Manica, M., Prasanna, P., Ballah, D., Shah, P.L., and Karteris, E. (2020). On the Role of Artificial Intelligence in Medical Imaging of COVID-19. medRxiv.10.1101/2020.09.02.20187096 ↗
41. Deep learning in medical ultrasound analysis: A review10.1016/j.eng.2018.11.020 ↗
42. Localizing B-lines in lung ultrasonography by weakly-supervised deep learning, in-vivo results
43. Quantifying lung ultrasound comets with a convolutional neural network: Initial clinical results10.1016/j.compbiomed.2019.02.002 ↗
44. Ultrasound-Based Detection of Lung Abnormalities Using Single Shot Detection Convolutional Neural Networks10.1007/978-3-030-01045-4_8 ↗
45. Automatic pleural line extraction and COVID-19 scoring from lung ultrasound data10.1109/TUFFC.2020.3005512 ↗
46. Boundary Restored Network for Subpleural Pulmonary Lesion Segmentation on Ultrasound Images at Local and Global Scales10.1007/s10278-020-00356-8 ↗
47. Computer-aided diagnosis of endobronchial ultrasound images using convolutional neural network10.1016/j.cmpb.2019.05.020 ↗
48. Born, J., Brändle, G., Cossio, M., Disdier, M., Goulet, J., Roulin, J., and Wiedemann, N. (2020). POCOVID-Net: Automatic Detection of COVID-19 From a New Lung Ultrasound Imaging Dataset (POCUS). arXiv.
49. Roberts, J., and Tsiligkaridis, T. (2020). Ultrasound Diagnosis of COVID-19: Robustness and Explainability. arXiv.
50. Chen, Y., Zhang, C., Liu, L., Feng, C., Dong, C., Luo, Y., and Wan, X. (2020). Effective Sample Pair Generation for Ultrasound Video Contrastive Representation Learning. arXiv.
51. Hou, D., Hou, R., and Hou, J. (2020, January 28–31). Interpretable Saab Subspace Network for COVID-19 Lung Ultrasound Screening. Proceedings of the 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, USA.10.1109/UEMCON51285.2020.9298069 ↗
52. A novel statistical approach for multiplicative speckle removal using t-locations scale and non-sub sampled shearlet transform10.1016/j.dsp.2020.102857 ↗
53. Baum, Z., Bonmati, E., Cristoni, L., Walden, A., Prados, F., Kanber, B., Barratt, D.C., Hawkes, D.J., Parker, G.J., and Wheeler-Kingshott, C.A. (2020). Image quality assessment for closed-loop computer-assisted lung ultrasound. arXiv.10.1117/12.2581865 ↗
54. Liu, L., Lei, W., Luo, Y., Feng, C., Wan, X., and Liu, L. (2020). Semi-Supervised Active Learning for COVID-19 Lung Ultrasound Multi-symptom Classification. arXiv.10.1109/ICTAI50040.2020.00191 ↗
55. Zhang, J., Chng, C.B., Chen, X., Wu, C., Zhang, M., Xue, Y., Jiang, J., and Chui, C.K. (2020, January 13–15). Detection and Classification of Pneumonia from Lung Ultrasound Images. Proceedings of the 2020 5th International Conference on Communication, Image and Signal Processing (CCISP), Chengdu, China.10.1109/CCISP51026.2020.9273469 ↗
56. Arntfield, R., VanBerlo, B., Alaifan, T., Phelps, N., White, M., Chaudhary, R., Ho, J., and Wu, D. (2020). Development of a deep learning classifier to accurately distinguish COVID-19 from look-a-like pathology on lung ultrasound. medRxiv.10.1101/2020.10.13.20212258 ↗
57. Deep learning for classification and localization of COVID-19 markers in point-of-care lung ultrasound10.1109/TMI.2020.2994459 ↗
58. Assessment of COVID-19 in lung ultrasound by combining anatomy and sonographic artifacts using deep learning10.1121/1.5147600 ↗
59. Yaron, D., Keidar, D., Goldstein, E., Shachar, Y., Blass, A., Frank, O., Schipper, N., Shabshin, N., Grubstein, A., and Suhami, D. (2020). Point of Care Image Analysis for COVID-19. arXiv.10.1109/ICASSP39728.2021.9413687 ↗
60. Jim, A.A.J., Rafi, I., Chowdhury, M.S., Sikder, N., Mahmud, M.P., Rubaie, S., Masud, M., Bairagi, A.K., Bhakta, K., and Nahid, A.A. (2020). An Automatic Computer-Based Method for Fast and Accurate Covid-19 Diagnosis. medRxiv.
61. Is there a role for lung ultrasound during the COVID-19 pandemic?10.1002/jum.15284 ↗
62. Lichtenstein, D.A. (2015). Lung Ultrasound in the Critically Ill: The BLUE Protocol, Springer.10.1007/978-3-319-15371-1 ↗
63. Jackson, K., Butler, R., and Aujayeb, A. (2020). Lung ultrasound in the COVID-19 pandemic. Postgrad. Med J., 97.10.1136/postgradmedj-2020-138137 ↗
64. Consolidating medical ambulatory care services in the COVID-19 era
65. Consolidating malignant pleural and peritoneal services during the COVID-19 response10.7861/fhj.2020-0016 ↗
66. Comparison of the Computed Tomography Findings in COVID-19 and Other Viral Pneumonia in Immunocompetent Adults: A Systematic Review and Meta-Analysis10.1007/s00330-020-07018-x ↗
67. Current Concepts Imaging in COVID-19 and the Challenges for Low and Middle Income Countries
68. Imagenet large scale visual recognition challenge10.1007/s11263-015-0816-y ↗
69. Dataset, K. (2020, May 10). Ultrasound Nerve Segmentation, 206. Data Retrieved from Kaggle. Available online: https://www.kaggle.com/c/ultrasound-nerve-segmentation.
70. Simonyan, K., and Zisserman, A. (2015, January 7–9). Very Deep Convolutional Networks for Large-Scale Image Recognition. Proceedings of the 3rd International Conference on Learning Representations, San Diego, CA, USA.
71. Zoph, B., Vasudevan, V., Shlens, J., and Le, Q.V. (2018, January 18–23). Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00907 ↗
72. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., and Torralba, A. (2016, January 27–30). Learning deep features for discriminative localization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.319 ↗
73. Zhou, Z., Sodha, V., Rahman Siddiquee, M.M., Feng, R., Tajbakhsh, N., Gotway, M.B., and Liang, J. (2019). Medical Image Computing and Computer Assisted Intervention–MICCAI 2019, Springer International Publishing.
74. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
75. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. (2017, January 22–29). Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.74 ↗
76. A kernel two-sample test
77. Gal, Y., and Ghahramani, Z. (2016, January 19–24). Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA.
78. Ayhan, M.S., and Berens, P. (2018, January 4). Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks. Proceedings of the 1st Conference on Medical Imaging with Deep Learning (MIDL 2018), Amsterdam, The Netherlands.
79. Butt, C., Gill, J., Chun, D., and Babu, B.A. (2020). Deep learning system to screen coronavirus disease 2019 pneumonia. Appl. Intell.10.1007/s10489-020-01714-3 ↗
80. Barish, M., Bolourani, S., Lau, L.F., Shah, S., and Zanos, T.P. (2020). External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nat. Mach. Intell.10.1038/s42256-020-00254-2 ↗
81. Vedula, S., Senouf, O., Bronstein, A.M., Michailovich, O.V., and Zibulevsky, M. (2017). Towards CT-quality Ultrasound Imaging using Deep Learning. arXiv.
82. Zhang, L., Vishnevskiy, V., and Goksel, O. (2020). Deep Network for Scatterer Distribution Estimation for Ultrasound Image Simulation. arXiv.10.1109/TUFFC.2020.3018424 ↗
83. Clinical Review: Point-of-Care Ultrasound in the Evaluation of COVID-1910.1016/j.jemermed.2020.06.032 ↗
84. Sonographic signs and patterns of COVID-19 pneumonia10.1186/s13089-020-00171-w ↗
85. A proposed lung ultrasound and phenotypic algorithm for the care of COVID-19 patients with acute respiratory failure10.1007/s12630-020-01704-6 ↗
86. The Diagnosis of Pneumonia in a Pregnant Woman with COVID-19 Using Maternal Lung Ultrasound10.1016/j.ajog.2020.04.020 ↗
87. Huang, Y., Wang, S., Liu, Y., Zhang, Y., Zheng, C., Zheng, Y., Zhang, C., Min, W., Zhou, H., and Yu, M. (2020, March 28). A Preliminary Study on the Ultrasonic Manifestations of Peripulmonary Lesions of Non-Critical Novel Coronavirus Pneumonia (COVID-19). Available online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3544750.10.2139/ssrn.3544750 ↗
88. Advances in point-of-care thoracic ultrasound10.1016/j.emc.2015.09.003 ↗
89. Clinical review: Bedside lung ultrasound in critical care practice10.1186/cc5668 ↗
90. COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: Single-center study and comprehensive radiologic literature review10.1016/j.ejro.2020.100231 ↗
91. Early recognition of the 2009 pandemic influenza A (H1N1) pneumonia by chest ultrasound10.1186/cc11201 ↗
92. Yassa, M., Birol, P., Mutlu, A.M., Tekin, A.B., Sandal, K., and Tug, N. (2020). Lung Ultrasound Can Influence the Clinical Treatment of Pregnant Women with COVID-19. J. Ultrasound Med., 40.10.1002/jum.15367 ↗
93. Lung ultrasound for the diagnosis of community-acquired pneumonia in children10.1007/s00247-017-3910-1 ↗
94. Lung ultrasound in community-acquired pneumonia and in interstitial lung diseases10.1159/000357449 ↗
95. Prospective application of clinician-performed lung ultrasonography during the 2009 H1N1 influenza A pandemic: Distinguishing viral from bacterial pneumonia10.1186/2036-7902-4-16 ↗
96. Role of point-of-care ultrasound during the COVID-19 pandemic: Our recommendations in the management of dialytic patients10.1186/s13089-020-00177-4 ↗
97. Thoracic ultrasound and SARS-COVID-19: A pictorial essay10.1007/s40477-020-00458-7 ↗
98. Usefulness of lung ultrasound in diagnosing causes of exacerbation in patients with chronic dyspnea10.5603/PiAP.a2015.0083 ↗