Improving Accuracy of Contactless Respiratory Rate Estimation by Enhancing Thermal Sequences with Deep Neural Networks
Source abstract
Abstract
Estimation of vital signs using image processing techniques have already been proved to have a potential for supporting remote medical diagnostics and replacing traditional measurements that usually require special hardware and electrodes placed on a body. In this paper, we further extend studies on contactless Respiratory Rate (RR) estimation from extremely low resolution thermal imagery by enhancing acquired sequences using Deep Neural Networks (DNN). To perform extensive benchmark evaluation, we acquired two thermal datasets using FLIR® cameras with a spatial resolution of 80 × 60 and 320 × 240 from 71 volunteers in total. In-depth analysis of the proposed Convolutional-based Super Resolution model showed that for images downscaled with a factor of 2 and then super-resolved using Deep Learning (DL) can lead to better RR estimation accuracy than from original high-resolution sequences. In addition, if an estimator based on a dominating peak in the frequency domain is used, SR can outperform original data for a down-scale factor of 4 and images as small as 20 × 15 pixels. Our study also showed that RR estimation accuracy is better for super-resolved data than for images with color changes magnified using algorithms previously applied in the literature for enhancing vital signs patterns.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Non-Invasive Vital Sign Monitoring · Heart Rate Variability and Autonomic Control · Optical Imaging and Spectroscopy Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Biomedical Engineering, Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Gabriela Narutowicza 11/12, 80-233 Gdansk, Poland
Artificial Intelligence Products Group, Intel Corporation, 12220 Scripps Summit Dr, San Diego, CA 92131, USA
References (54)
Show source-provided references
1. Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., and Song, L. (2017, January 21–26). Sphereface: Deep hypersphere embedding for face recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.713 ↗
2. Schroff, F., Kalenichenko, D., and Philbin, J. (2015, January 7–12). Facenet: A unified embedding for face recognition and clustering. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7298682 ↗
3. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016). Ssd: Single shot multibox detector. European Conference on Computer Vision, Springer.10.1007/978-3-319-46448-0_2 ↗
4. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
5. Qi, C.R., Su, H., Mo, K., and Guibas, L.J. (2017, January 21–26). Pointnet: Deep learning on point sets for 3d classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.
6. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., and Wang, Z. (2017, January 21–26). Photo-realistic single image super-resolution using a generative adversarial network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.19 ↗
7. Chaichulee, S., Villarroel, M., Jorge, J., Arteta, C., Green, G., McCormick, K., Zisserman, A., and Tarassenko, L. (June, January 30). Multi-task convolutional neural network for patient detection and skin segmentation in continuous non-contact vital sign monitoring. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.10.1109/FG.2017.41 ↗
8. Wang, Z., Yang, Z., and Dong, T. (2017). A review of wearable technologies for elderly care that can accurately track indoor position, recognize physical activities and monitor vital signs in real time. Sensors, 17.10.3390/s17020341 ↗
9. Neonatal non-contact respiratory monitoring based on real-time infrared thermography10.1186/1475-925X-10-93 ↗
10. Is thermal scanner losing its bite in mass screening of fever due to SARS?10.1118/1.1819532 ↗
11. Kwasniewska, A., Ruminski, J., Szankin, M., and Czuszynski, K. (2018, January 17–21). Remote Estimation of Video-Based Vital Signs in Emotion Invocation Studies. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.10.1109/EMBC.2018.8513423 ↗
12. Fei, J., Zhu, Z., and Pavlidis, I. (2006, January 17–18). Imaging breathing rate in the co 2 absorption band. Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China.
13. Non-contact monitoring of respiratory function using infrared imaging10.1109/MEMB.2006.1636352 ↗
14. Thermal infrared imaging: A novel method to monitor airflow during polysomnography10.1093/sleep/32.11.1521 ↗
15. Thermistor at a distance: Unobtrusive measurement of breathing
16. Rumiński, J. (2016, January 4–8). Evaluation of the respiration rate and pattern using a portable thermal camera. Proceedings of the 13th Quantitative Infrared Thermography Conference, Gdansk, Poland.10.21611/qirt.2016.107 ↗
17. Reliability of pulse measurements in videoplethysmography10.1515/mms-2016-0040 ↗
18. Spatiotemporal smoothing as a basis for facial tissue tracking in thermal imaging10.1109/TBME.2012.2232927 ↗
19. Al-Khalidi, F.Q., Saatchi, R., Burke, D., and Elphick, H. (2010, January 16–19). Tracking human face features in thermal images for respiration monitoring. Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications-AICCSA 2010, Hammamet, Tunisia.10.1109/AICCSA.2010.5586994 ↗
20. Contact-free respiration rate monitoring using a pan–tilt thermal camera for stationary bike telerehabilitation sessions10.1109/JSYST.2014.2336372 ↗
21. (2019, August 24). Flir Lepton Camera Modules. Available online: https://www.flir.com/products/lepton/.
22. Eulerian Video Magnification for Revealing Subtle Changes in the World10.1145/2185520.2185561 ↗
23. Phase-based video motion processing10.1145/2461912.2461966 ↗
24. Oh, T.H., Jaroensri, R., Kim, C., Elgharib, M., Durand, F., Freeman, W.T., and Matusik, W. (2018, January 8–14). Learning-based video motion magnification. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01225-0_39 ↗
25. Estrada, M., and Stowers, A. (2019, September 30). Amplification of Heart Rate in Multi-Subject Videos. Available online: Https://web.stanford.edu/class/ee368/Project_Spring_1415/Reports/Stowers_Estrada.pdf.
26. Balakrishnan, G., Durand, F., and Guttag, J. (2013, January 23–28). Detecting Pulse from Head Motions in Video. Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, Portland, OR, USA.10.1109/CVPR.2013.440 ↗
27. Aubakir, B., Nurimbetov, B., Tursynbek, I., and Varol, H.A. (2016, January 16–20). Vital sign monitoring utilizing Eulerian video magnification and thermography. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.10.1109/EMBC.2016.7591489 ↗
28. Bennett, S.L., Goubran, R., and Knoefel, F. (2017, January 11–15). Comparison of motion-based analysis to thermal-based analysis of thermal video in the extraction of respiration patterns. Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Seogwipo, Korea.10.1109/EMBC.2017.8037693 ↗
29. Images super-resolution using deep convolutional networks10.1109/TPAMI.2015.2439281 ↗
30. Kim, J., Kwon Lee, J., and Mu Lee, K. (2016, January 27–30). Deeply-recursive convolutional network for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.181 ↗
31. Tai, Y., Yang, J., and Liu, X. (2017, January 21–26). Image super-resolution via deep recursive residual network. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.298 ↗
32. Zhang, X., Li, C., Meng, Q., Liu, S., Zhang, Y., and Wang, J. (2018). Infrared Image Super Resolution by Combining Compressive Sensing and Deep Learning. Multidiscip. Digit. Publ. Inst. Sens. J., 8.10.3390/s18082587 ↗
33. Multi-Scale Residual Hierarchical Dense Networks for Single Image Super-Resolution10.1109/ACCESS.2019.2915943 ↗
34. Wang, X., Yu, K., Dong, C., and Change Loy, C. (2018, January 18–22). Recovering realistic texture in image super-resolution by deep spatial feature transform. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00070 ↗
35. Bulat, A., and Tzimiropoulos, G. (2018, January 18–22). Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA.10.1109/CVPR.2018.00019 ↗
36. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T., and Zelnik-Manor, L. (2018, January 8–14). The 2018 PIRM challenge on perceptual image super-resolution. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-11021-5_21 ↗
37. Cho, Y., Bianchi-Berthouze, N., Marquardt, N., and Julier, S.J. (2018, January 21–26). Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, Montreal, QC, Canada.10.1145/3173574.3173576 ↗
38. THERMALNET: A Deep Convolutional Network for Synthetic Thermal Image Generation10.5194/isprs-archives-XLII-2-W4-41-2017 ↗
39. Infrared Image Enhancement in Maritime Environment with Convolutional Neural Networks
40. Single infrared image enhancement using a deep convolutional neural network10.1016/j.neucom.2018.11.081 ↗
41. Cascaded Deep Networks with Multiple Receptive Fields for Infrared Image Super-Resolution10.1109/TCSVT.2018.2864777 ↗
42. Zhang, X., Li, C., Meng, Q., Liu, S., Zhang, Y., and Wang, J. (2018). Infrared Image Super Resolution by Combining Compressive Sensing and Deep Learning. Sensors, 18.10.3390/s18082587 ↗
43. Kwasniewska, A., Szankin, M., Ruminski, J., and Kaczmarek, M. (2019, January 23–27). Evaluating Accuracy of Respiratory Rate Estimation from SuperResolved Thermal Imagery. Proceedings of the IEEE EMBC Conference, Berlin, Germany.10.1109/EMBC.2019.8857764 ↗
44. Ruminski, J., and Kwasniewska, A. (2017). Evaluation of respiration rate using thermal imaging in mobile conditions. Application of Infrared to Biomedical Sciences, Springer.10.1007/978-981-10-3147-2_18 ↗
45. Multimodal platform for continuous monitoring of the elderly and disabled10.1166/jmihi.2012.1061 ↗
46. Ruminski, J., Bujnowski, A., Czuszynski, K., and Kocejko, T. (2016, January 11–14). Estimation of respiration rate using an accelerometer and thermal camera in eGlasses. Proceedings of the 2016 Federated Conference on Computer Science and Information Systems (FedCSIS), Gdańsk, Poland.10.15439/2016F329 ↗
47. Big data significance in remote medical diagnostics based on deep learning techniques
48. Kim, J., Kwon Lee, J., and Mu Lee, K. (2016, January 27–30). Accurate image super-resolution using very deep convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.182 ↗
49. He, K., Zhang, X., Ren, S., and Sun, J. (2016, January 27–30). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.90 ↗
50. (2019, August 10). Eulerian Video Magnification Online Tool. Available online: https://lambda.qrilab.com/site/.
51. Ruminski, J., Kwasniewska, A., Szankin, M., Kocejko, T., and Mazur-Milecka, M. (2019, January 23–27). Evaluation of Facial Pulse Signals Using Deep Neural Net Models. Proceedings of the IEEE EMBC Conference, Berlin, Germany.10.1109/EMBC.2019.8857839 ↗
52. Hochhausen, N., Barbosa Pereira, C., Leonhardt, S., Rossaint, R., and Czaplik, M. (2018). Estimating Respiratory Rate in Post-Anesthesia Care Unit Patients Using Infrared Thermography: An Observational Study. Sensors, 18.10.3390/s18051618 ↗
53. Alam, S., Singh, S.P., and Abeyratne, U. (2017, January 11–15). Considerations of handheld respiratory rate estimation via a stabilized Video Magnification approach. Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju Island, Korea.10.1109/EMBC.2017.8037805 ↗
54. Szankin, M., Kwasniewska, A., Sirlapu, T., Wang, M., Ruminski, J., Nicolas, R., and Bartscherer, M. (2018, January 17–21). Long Distance Vital Signs Monitoring with Person Identification for Smart Home Solutions. Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA.10.1109/EMBC.2018.8512509 ↗