DataVXApplied SciencesPaper
journal article·2023-01-18Open access

Inter-Rater Variability in the Evaluation of Lung Ultrasound in Videos Acquired from COVID-19 Patients

Joaquin L. Herraiz, Clara Freijo, Jorge Camacho, Mario Muñoz, Ricardo González, Rafael Alonso-Roca, Jorge Álvarez-Troncoso, Luis Matías Beltrán-Romero, Máximo Bernabeu-Wittel, Rafael Blancas, Antonio Calvo-Cebrián, Ricardo Campo-Linares, Jaldún Chehayeb-Morán, Jose Chorda-Ribelles, Samuel García-Rubio, Gonzalo García-de-Casasola, Adriana Gil-Rodrigo, César Henríquez-Camacho, Alba Hernandez-Píriz, Carlos Hernandez-Quiles, Rafael Llamas-Fuentes, Davide Luordo, Raquel Marín-Baselga, María Cristina Martínez-Díaz, María Mateos-González, Manuel Mendez-Bailon, Francisco Miralles-Aguiar, Ramón Nogue, Marta Nogué, Borja Ortiz de Urbina-Antia, Alberto Ángel Oviedo-García, José M. Porcel, Santiago Rodriguez, Diego Aníbal Rodríguez-Serrano, Talía Sainz, Ignacio Manuel Sánchez-Barrancos, Marta Torres-Arrese, Juan Torres-Macho, Angela Trueba Vicente, Tomas Villén-Villegas, Juan José Zafra-Sánchez, Yale Tung-Chen

Applied Sciences· Volume 13· Issue 3
Publisher pageOpen full text
Source abstract

Abstract

Lung ultrasound (LUS) allows for the detection of a series of manifestations of COVID-19, such as B-lines and consolidations. The objective of this work was to study the inter-rater reliability (IRR) when detecting signs associated with COVID-19 in the LUS, as well as the performance of the test in a longitudinal or transverse orientation. Thirty-three physicians with advanced experience in LUS independently evaluated ultrasound videos previously acquired using the ULTRACOV system on 20 patients with confirmed COVID-19. For each patient, 24 videos of 3 s were acquired (using 12 positions with the probe in longitudinal and transverse orientations). The physicians had no information about the patients or other previous evaluations. The score assigned to each acquisition followed the convention applied in previous studies. A substantial IRR was found in the cases of normal LUS (κ = 0.74), with only a fair IRR for the presence of individual B-lines (κ = 0.36) and for confluent B-lines occupying < 50% (κ = 0.26) and a moderate IRR in consolidations and B-lines > 50% (κ = 0.50). No statistically significant differences between the longitudinal and transverse scans were found. The IRR for LUS of COVID-19 patients may benefit from more standardized clinical protocols.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Ultrasound in Clinical Applications · Phonocardiography and Auscultation Techniques · Radiology practices and education

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Nuclear Physics Group, EMFTEL and IPARCOS, Universidad Complutense de Madrid, 28040 Madrid, Spain

Health Research Institute (IdISSC), Hospital Clinico San Carlos, 28040 Madrid, Spain

Group of Ultrasound Systems and Technologies, Institute of Physical and Information Technologies (ITEFI), Spanish National Research Council (CSIC), 28040 Madrid, Spain

Dasel SL, Arganda del Rey, 28500 Madrid, Spain

Centro de Salud Mar Báltico, 28040 Madrid, Spain

Internal Medicine Department, Hospital University La Paz, 28046 Madrid, Spain

Internal Medicine Department, Hospital Virgen del Rocío, 41013 Sevilla, Spain

Department of Medicine, University of Seville, 41009 Seville, Spain

Department of Medicine, Universidad Alfonso X El Sabio. Intensive Care Unit, Hospital Universitario del Tajo, 28300 Aranjuez, Spain

Centro de Salud de Robledo de Chavela, 28294 Robledo de Chavela, Spain

Emergency Department, Hospital Santa Bárbara, 13500 Puertollano, Spain

Emergency Department, Hospital Universitario Clínico de Valladolid, 47003 Valladolid, Spain

Internal Medicine Department, Hospital Universitario General de Valencia, 46014 Valencia, Spain

Internal Medicine Department, Hospital Santa Marina, 48004 Bilbao, Spain

Emergency Department, Hospital Fundación de Alcorcón, 28922 Madrid, Spain

Emergency Department, Dr. Balmis General University Hospital, Alicante Institute for Health and Biomedical Research (ISABIAL), 03010 Alicante, Spain

Internal Medicine Department, Hospital Universitario Rey Juan Carlos, 28933 Móstoles, Spain

Internal Medicine Department, Hospital Infanta Cristina, 28981 Parla, Spain

Emergency Department, Hospital Reina Sofia, 14004 Córdoba, Spain

Intensive Care Medicine Department, Hospital Universitario Príncipe de Asturias, 28805 Alcalá de Henares, Spain

Internal Medicine Department, Hospital Universitario Clínico San Carlos, 28040 Madrid, Spain

Anesthesiology Department, Hospital Universitario Puerta del Mar, 11009 Cádiz, Spain

Department of Medicine, Universitat de Lleida, 25008 Lleida, Spain

Pneumology Department, Hospital Universitario de Cruces, 48903 Barakaldo, Spain

Emergency Department, Hospital de Valme, 41014 Sevilla, Spain

Funding

Centre for Industrial Technological Development · Google Cloud Research Credits program · MCIU/AEI/FEDER UE and by the European Commission–NextGenerationEU · European Commission–NextGenerationEU

References (23)

Show source-provided references
1. New International Guidelines and Consensus on the Use of Lung Ultrasound10.1002/jum.16088 ↗
2. Lung Ultrasound Integration in Assessment of Patients with Noncritical COVID-19
3. Torres-Macho, J., Sánchez-Fernández, M., Arnanz-González, I., Tung-Chen, Y., Franco-Moreno, A.I., Duffort-Falcó, M., Beltrán-Romero, L., Rodríguez-Suaréz, S., Bernabeu-Wittel, M., and Urbano, E. (2021). Prediction Accuracy of Serial Lung Ultrasound in COVID-19 Hospitalized Patients (Pred-Echovid Study). J. Clin. Med., 10.10.3390/jcm10214818 ↗
4. Lung Ultrasound for the Early Diagnosis of COVID-19 Pneumonia: An International Multicenter Study10.1007/s00134-021-06373-7 ↗
5. Usefulness of Lung Ultrasound Examinations Performed by Primary Care Physicians in Patients With Suspected COVID-1910.1002/jum.15444 ↗
6. Thoracic Imaging Tests for the Diagnosis of COVID-19
7. Early Lung Ultrasound Findings in Patients With COVID-19 Pneumonia: A Retrospective Multicenter Study of 479 Patients
8. Artificial Intelligence and Democratization of the Use of Lung Ultrasound in COVID-19: On the Feasibility of Automatic Calculation of Lung Ultrasound Score
9. Correlation between Chest Computed Tomography and Lung Ultrasonography in Patients with Coronavirus Disease 2019 (COVID-19)10.1016/j.ultrasmedbio.2020.07.003 ↗
10. Pleural Diseases and COVID-19: Ubi Fumus, Ibi Ignis10.1183/13993003.03308-2020 ↗
11. Hernández-Píriz, A., Tung-Chen, Y., Jiménez-Virumbrales, D., Ayala-Larrañaga, I., Barba-Martín, R., Canora-Lebrato, J., Zapatero-Gaviria, A., and Casasola-Sánchez, G.G.D. (2021). Importance of Lung Ultrasound Follow-Up in Patients Who Had Recovered from Coronavirus Disease 2019: Results from a Prospective Study. J. Clin. Med., 10.10.3390/jcm10143196 ↗
12. The lung ultrasound “Rule of 7” in the prognosis of COVID-19 patients: Results from a prospective multicentric study
13. Multi-Organ Point-of-Care Ultrasound for COVID-19 (PoCUS4COVID): International Expert Consensus10.1186/s13054-020-03369-5 ↗
14. Interobserver Agreement of Lung Ultrasound Findings of COVID-19
15. The Inter-Rater Reliability of Pediatric Point-of-Care Lung Ultrasound Interpretation in Children With Acute Respiratory Failure
16. Fatima, N., Mento, F., Zanforlin, A., Smargiassi, A., Torri, E., Perrone, T., and Demi, L. (2022). Human-to-AI Interrater Agreement for Lung Ultrasound Scoring in COVID-19 Patients. J. Ultrasound Med. Off. J. Am. Inst. Ultrasound Med.10.1002/jum.16052 ↗
17. Inter-Observer Reliability for Different Point-of-Care Lung Ultrasound Findings in Mechanically Ventilated Critically Ill COVID-19 Patients10.1007/s10877-021-00726-9 ↗
18. The Impact of Different Lung Ultrasound Protocols in the Assessment of Lung Lesions in COVID-19 Patients: Is There an Ideal Lung Ultrasound Protocol?10.1007/s40477-021-00610-x ↗
19. Interrater Reliability: The Kappa Statistic10.11613/BM.2012.031 ↗
20. A Comparison of Consensus, Consistency, and Measurement Approaches to Estimating Interrater Reliability
21. Mateos González, M., García de Casasola Sánchez, G., Muñoz, F.J.T., Proud, K., Lourdo, D., Sander, J.-V., Jaimes, G.E.O., Mader, M., Canora Lebrato, J., and Restrepo, M.I. (2021). Comparison of Lung Ultrasound versus Chest X-ray for Detection of Pulmonary Infiltrates in COVID-19. Diagnostics, 11.10.3390/diagnostics11020373 ↗
22. Pellegrino, F., Carnevale, A., Bisi, R., Cavedagna, D., Reverberi, R., Uccelli, L., Leprotti, S., and Giganti, M. (2022). Best Practices on Radiology Department Workflow: Tips from the Impact of the COVID-19 Lockdown on an Italian University Hospital. Healthcare, 10.10.3390/healthcare10091771 ↗
23. Wang, M., Luo, X., Wang, L., Estill, J., Lv, M., Zhu, Y., Wang, Q., Xiao, X., Song, Y., and Lee, M.S. (2021). A Comparison of Lung Ultrasound and Computed Tomography in the Diagnosis of Patients with COVID-19: A Systematic Review and Meta-Analysis. Diagnostics, 11.10.3390/diagnostics11081351 ↗