Real-Time Webcam Heart-Rate and Variability Estimation with Clean Ground Truth for Evaluation
Source abstract
Abstract
Remote photo-plethysmography (rPPG) uses a camera to estimate a person’s heart rate (HR). Similar to how heart rate can provide useful information about a person’s vital signs, insights about the underlying physio/psychological conditions can be obtained from heart rate variability (HRV). HRV is a measure of the fine fluctuations in the intervals between heart beats. However, this measure requires temporally locating heart beats with a high degree of precision. We introduce a refined and efficient real-time rPPG pipeline with novel filtering and motion suppression that not only estimates heart rates, but also extracts the pulse waveform to time heart beats and measure heart rate variability. This unsupervised method requires no rPPG specific training and is able to operate in real-time. We also introduce a new multi-modal video dataset, VicarPPG 2, specifically designed to evaluate rPPG algorithms on HR and HRV estimation. We validate and study our method under various conditions on a comprehensive range of public and self-recorded datasets, showing state-of-the-art results and providing useful insights into some unique aspects. Lastly, we make available CleanerPPG, a collection of human-verified ground truth peak/heart-beat annotations for existing rPPG datasets. These verified annotations should make future evaluations and benchmarking of rPPG algorithms more accurate, standardized and fair.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Non-Invasive Vital Sign Monitoring · Heart Rate Variability and Autonomic Control · Optical Imaging and Spectroscopy Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Vicarious Perception Technologies (VicarVision), 1015 AH Amsterdam, The Netherlands
Intelligent Systems, Delft University of Technology, 2628 XE Delft, The Netherlands
Funding
Nederlandse Organisatie voor Wetenschappelijk Onderzoek
References (64)
Show source-provided references
1. Photoplethysmography and its application in clinical physiological measurement10.1088/0967-3334/28/3/R01 ↗
2. Heart rate estimation using facial video: A review10.1016/j.bspc.2017.07.004 ↗
3. Remote plethysmographic imaging using ambient light10.1364/OE.16.021434 ↗
4. Gudi, A., Bittner, M., Lochmans, R., and van Gemert, J. (2019, January 27–28). Efficient Real-Time Camera Based Estimation of Heart Rate and Its Variability. Proceedings of the IEEE International Conference on Computer Vision Workshops, Seoul, Korea.10.1109/ICCVW.2019.00196 ↗
5. Chen, W., and McDuff, D. (2018). Deepphys: Video-based physiological measurement using convolutional attention networks. Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018, Springer.10.1007/978-3-030-01216-8_22 ↗
6. Spetlik, R., Cech, J., Franc, V., and Matas, J. (2018, January 3–6). Visual Heart Rate Estimation with Convolutional Neural Network. Proceedings of the British Machine Vision Conference, Newcastle upon Tyne, UK.
7. Eulerian Video Magnification for Revealing Subtle Changes in the World10.1145/2185520.2185561 ↗
8. Zhang, Y., Pintea, S.L., and Van Gemert, J.C. (2017, January 21–26). Video acceleration magnification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.61 ↗
9. Kleiger, R.E., Stein, P.K., and Bigger, J.T. (2005). Heart Rate Variability: Measurement and Clinical Utility. Ann. Noninvasive Electrocardiol., 88–101.10.1111/j.1542-474X.2005.10101.x ↗
10. An overview of heart rate variability metrics and norms10.3389/fpubh.2017.00258 ↗
11. A multimodal database for affect recognition and implicit tagging10.1109/T-AFFC.2011.25 ↗
12. DEAP: A Database for Emotion Analysis; Using Physiological Signals10.1109/T-AFFC.2011.15 ↗
13. Remote heart rate measurement using low-cost RGB face video: A technical literature review10.1007/s11704-016-6243-6 ↗
14. Main artifacts in electrocardiography10.1111/anec.12494 ↗
15. Photoplethysmography revisited: From contact to noncontact, from point to imaging10.1109/TBME.2015.2476337 ↗
16. Improvements in remote cardiopulmonary measurement using a five band digital camera10.1109/TBME.2014.2323695 ↗
17. Remote detection of photoplethysmographic systolic and diastolic peaks using a digital camera10.1109/TBME.2014.2340991 ↗
18. Advancements in noncontact, multiparameter physiological measurements using a webcam10.1109/TBME.2010.2086456 ↗
19. Robust pulse rate from chrominance-based rPPG10.1109/TBME.2013.2266196 ↗
20. Li, X., Chen, J., Zhao, G., and Pietikainen, M. (2014, January 23–28). Remote heart rate measurement from face videos under realistic situations. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.543 ↗
21. Tasli, H.E., Gudi, A., and den Uyl, M. (2014, January 27–30). Remote PPG based vital sign measurement using adaptive facial regions. Proceedings of the 2014 IEEE International Conference on Image Processing (ICIP), Paris, France.10.1109/ICIP.2014.7025282 ↗
22. Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., and Sebe, N. (2016, January 27–30). Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.263 ↗
23. Algorithmic principles of remote PPG10.1109/TBME.2016.2609282 ↗
24. An advanced detrending method with application to HRV analysis10.1109/10.979357 ↗
25. LeCun, Y., and Bengio, Y. (1998). Convolutional Networks for Images, Speech, and Time Series. The Handbook of Brain Theory and Neural Networks, MIT Press.
26. Rhythmnet: End-to-end heart rate estimation from face via spatial-temporal representation10.1109/TIP.2019.2947204 ↗
27. Yu, Z., Li, X., Niu, X., Shi, J., and Zhao, G. (2020). AutoHR: A Strong End-to-end Baseline for Remote Heart Rate Measurement with Neural Searching. arXiv.10.1109/LSP.2020.3007086 ↗
28. Van Kuilenburg, H., Wiering, M., and Den Uyl, M. (2005). A model based method for automatic facial expression recognition. European Conference on Machine Learning, Springer.10.1007/11564096_22 ↗
29. Heart rate variability extraction from videos signals: ICA vs. EVM comparison10.1109/ACCESS.2017.2678521 ↗
30. Noncontact imaging photoplethysmography to effectively access pulse rate variability10.1117/1.JBO.18.6.061205 ↗
31. Video pulse rate variability analysis in stationary and motion conditions10.1186/s12938-018-0437-0 ↗
32. Feasibility of assessing ultra-short-term pulse rate variability from video recordings10.7717/peerj.8342 ↗
33. Stricker, R., Müller, S., and Gross, H.M. (2014, January 25–29). Non-contact video-based pulse rate measurement on a mobile service robot. Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication, Edinburgh, UK.10.1109/ROMAN.2014.6926392 ↗
34. Li, P., Benezeth, Y., Nakamura, K., Gomez, R., Li, C., and Yang, F. (2019, January 19–21). An Improvement for Video-based Heart Rate Variability Measurement. Proceedings of the 2019 IEEE 4th International Conference on Signal and Image Processing (ICSIP), Wuxi, China.10.1109/SIPROCESS.2019.8868712 ↗
35. Song, R., Chen, H., Cheng, J., Li, C., Liu, Y., and Chen, X. (2020). PulseGAN: Learning to generate realistic pulse waveforms in remote photoplethysmography. arXiv.10.1109/JBHI.2021.3051176 ↗
36. Unsupervised skin tissue segmentation for remote photoplethysmography10.1016/j.patrec.2017.10.017 ↗
37. Measurement of heart rate variability using off-the-shelf smart phones10.1186/s12938-016-0127-8 ↗
38. VIPL-HR: A Multi-modal Database for Pulse Estimation from Less-Constrained Face Video10.1007/978-3-030-20873-8_36 ↗
39. Artemyev, M., Churikova, M., Grinenko, M., and Perepelkina, O. (2020). Robust algorithm for remote photoplethysmography in realistic conditions. Digit. Signal Process., 102737.10.1016/j.dsp.2020.102737 ↗
40. Viola, P., and Jones, M. (2001, January 8–14). Rapid object detection using a boosted cascade of simple features. Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Kauai, HI, USA.
41. Improved motion robustness of remote-PPG by using the blood volume pulse signature10.1088/0967-3334/35/9/1913 ↗
42. Heart rate variability: Standards of measurement, physiological interpretation, and clinical use10.1093/oxfordjournals.eurheartj.a014868 ↗
43. The use of fast Fourier transform for the estimation of power spectra: A method based on time averaging over short, modified periodograms10.1109/TAU.1967.1161901 ↗
44. Oy, K. (2020, May 05). About HRV. Available online: https://www.kubios.com/about-hrv.
45. Zhang, Z., Girard, J.M., Wu, Y., Zhang, X., Liu, P., Ciftci, U., Canavan, S., Reale, M., Horowitz, A., and Yang, H. (2016, January 27–30). Multimodal spontaneous emotion corpus for human behavior analysis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.374 ↗
46. Heusch, G., Anjos, A., and Marcel, S. (2017). A Reproducible Study on Remote Heart Rate Measurement. arXiv.
47. The validity and practicality of sun-reactive skin types I through VI10.1001/archderm.1988.01670060015008 ↗
48. Half a century of research on the Stroop effect: An integrative review10.1037/0033-2909.109.2.163 ↗
49. Billauer, E. (2020, March 13). Peakdet: Peak Detection Using MATLAB. Available online: http://billauer.co.il/peakdet.html.
50. Video-based human heart rate measurement using joint blind source separation10.1016/j.bspc.2016.08.020 ↗
51. Macwan, R., Bobbia, S., Benezeth, Y., Dubois, J., and Mansouri, A. (2018, January 18–22). Periodic variance maximization using generalized eigenvalue decomposition applied to remote photoplethysmography estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00181 ↗
52. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation10.1364/OE.18.010762 ↗
53. Demirezen, H., and Erdem, C.E. (2018, January 15–20). Remote photoplethysmography using nonlinear mode decomposition. Proceedings of the 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, AB, Canada.10.1109/ICASSP.2018.8462538 ↗
54. A novel algorithm for remote photoplethysmography: Spatial subspace rotation10.1109/TBME.2015.2508602 ↗
55. A wavelet-based decomposition method for a robust extraction of pulse rate from video recordings10.7717/peerj.5859 ↗
56. Heart rate estimation using remote photoplethysmography with multi-objective optimization10.1016/j.bspc.2018.10.012 ↗
57. Visual heart rate estimation and negative feedback control for fitness exercise10.1016/j.bspc.2019.101680 ↗
58. New insights on super-high resolution for video-based heart rate estimation with a semi-blind source separation method10.1016/j.compbiomed.2019.103535 ↗
59. Hu, M., Guo, D., Wang, X., Ge, P., and Chu, Q. (2019, January 19–21). A Novel Spatial-Temporal Convolutional Neural Network for Remote Photoplethysmography. Proceedings of the 2019 IEEE 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, Suzhou, China.10.1109/CISP-BMEI48845.2019.8966034 ↗
60. Sun, W., Wei, H., and Li, X. (2020, January 12–14). No-contact heart rate monitoring based on channel attention convolution model. Proceedings of the Eleventh International Conference on Graphics and Image Processing, International Society for Optics and Photonics, Hangzhou, China.10.1117/12.2557538 ↗
61. Tsou, Y.Y., Lee, Y.A., Hsu, C.T., and Chang, S.H. (April, January 30). Siamese-rPPG network: Remote photoplethysmography signal estimation from face videos. Proceedings of the 35th Annual ACM Symposium on Applied Computing, Brno, Czech Republic.
62. Yu, Z., Li, X., and Zhao, G. (2019, January 9–12). Remote Photoplethysmograph Signal Measurement from Facial Videos Using Spatio-Temporal Networks. Proceedings of the 30th British Machine Vision Conference 2019, Oxford, UK.
63. The FaceReader: Online facial expression recognition
64. Zhao, C., Lin, C.L., Chen, W., and Li, Z. (2018, January 18–22). A novel framework for remote photoplethysmography pulse extraction on compressed videos. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA.10.1109/CVPRW.2018.00177 ↗