MC-EVM: A Movement-Compensated EVM Algorithm with Face Detection for Remote Pulse Monitoring
Source abstract
Abstract
Automated tasks, mainly in the biomedical field, help to develop new technics to provide faster solutions for monitoring patients’ health status. For instance, they help to measure different types of human bio-signal, perform fast data analysis, and enable overall patient status monitoring. Eulerian Video Magnification (EVM) can reveal small-scale and hidden changes in real life such as color and motion changes that are used to detect actual pulse. However, due to patient movement during the measurement, the EVM process will result in the wrong estimation of the pulse. In this research, we provide a working prototype for effective artefact elimination using a face movement compensated EVM (MC-EVM) which aims to track the human face as the main Region Of Interest (ROI) and then use EVM to estimate the pulse. Our primary contribution lays on the development and training of two face detection models using TensorFlow Lite: the Single-Shot MultiBox Detector (SSD) and the EfficientDet-Lite0 models that are used based on the computational capabilities of the device in use. By employing one of these models, we can crop the face accurately from the video, which is then processed using EVM to estimate the pulse. MC-EVM showed very promising results and ensured robust pulse measurement by effectively mitigating the impact of patient movement. The results were compared and validated against ground-truth data that were made available online and against pre-existing solutions from the state-of-the-art.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Non-Invasive Vital Sign Monitoring · EEG and Brain-Computer Interfaces · Blind Source Separation Techniques
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
BioTech Research Center, Doctoral School of Applied Informatics and Applied Mathematics, Óbuda University, 1034 Budapest, Hungary
John Von Neumann Faculty of Informatics, Institute of Biomatics, Óbuda University, 1034 Budapest, Hungary
Applied Cyber-Medical Systems Research Team, Laboratory of Parallel and Distributed Systems, Institute for Computer Science and Control (SZTAKI), Hungarian Research Network (HUN-REN), 1518 Budapest, Hungary
Funding
Innovációs szolgáltató bázis létrehozása diagnosztikai, terápiás és kutatási célú kiberorvosi rendszerek fejlesztésére
References (54)
Show source-provided references
1. Eulerian video magnification for revealing subtle changes in the world10.1145/2185520.2185561 ↗
2. Revealing invisible changes in the world
3. Motion magnification10.1145/1073204.1073223 ↗
4. Yuenyongchaiwat, K., Sermsinsathong, N., Songsorn, P., Charususin, N., Buranapuntalug, S., Buekban, C., and Thanawattano, C. (2024). Development and feasibility assessment of a virtual reality-based aerobic exercise program with real-time pulse rate monitoring on hemodynamic and arterial stiffness in healthy people: A pilot study. Front. Digit. Health, 6.10.3389/fdgth.2024.1356837 ↗
5. Wang, Y., Wang, X., Nie, S., Meng, K., and Lin, Z. (2024). Recent Progress of Wearable Triboelectric Nanogenerator-Based Sensor for Pulse Wave Monitoring. Sensors, 24.10.3390/s24010036 ↗
6. Eulerian video magnification for heart pulse measurement in MRI scanners
7. Miljković, N., and Trifunović, D. (2014, January 25–27). Pulse rate assessment: Eulerian video magnification vs. electrocardiography recordings. Proceedings of the 12th Symposium on Neural Network Applications in Electrical Engineering (NEUREL), Belgrade, Serbia.10.1109/NEUREL.2014.7011447 ↗
8. He, X., Goubran, R.A., and Liu, X.P. (2014, January 11–12). Using Eulerian video magnification framework to measure pulse transit time. Proceedings of the 2014 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Lisboa, Portugal.10.1109/MeMeA.2014.6860029 ↗
9. He, X., Goubran, R.A., and Liu, X.P. (2016, January 24–27). Wrist pulse measurement and analysis using Eulerian video magnification. Proceedings of the 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), Las Vegas, NV, USA.10.1109/BHI.2016.7455830 ↗
10. Alzahrani, A., and Whitehead, A. (2015, January 3–5). Preprocessing realistic video for contactless heart rate monitoring using video magnification. Proceedings of the 2015 12th Conference on Computer and Robot Vision, Washington, DC, USA.10.1109/CRV.2015.41 ↗
11. A novel noninvasive method for remote heart failure monitoring: The EuleriAn video Magnification applications In heart Failure studY (AMPLIFY)10.1038/s41746-019-0159-0 ↗
12. Aubakir, B., Nurimbetov, B., Tursynbek, I., and Varol, H.A. (2016, January 16–20). Vital sign monitoring utilizing Eulerian video magnification and thermography. Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA.10.1109/EMBC.2016.7591489 ↗
13. Janatka, M., Ramdoo, K.S., Tatla, T., Pachtrachai, K., Elson, D.S., and Stoyanov, D. (2017, January 13–16). Examining in vivo tympanic membrane mobility using smart phone video-otoscopy and phase-based Eulerian video magnification. Proceedings of the Medical Imaging 2017: Computer-Aided Diagnosis, International Society for Optics and Photonics, Orlando, FL, USA.10.1117/12.2253729 ↗
14. Bennett, S.L., Goubran, R., and Knoefel, F. (2016, January 15–18). Adaptive eulerian video magnification methods to extract heart rate from thermal video. Proceedings of the 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Benevento, Italy.10.1109/MeMeA.2016.7533818 ↗
15. Efficient denoising approach based eulerian video magnification for colour and motion variations
16. Ramesh, R. (2021). The Computerized Classification of Micro-Motions in the Hand Using Waveforms from Mobile Phone. arXiv.
17. Leibe, B., Matas, J., Sebe, N., and Welling, M. (2016). SSD: Single Shot MultiBox Detector. Computer Vision—ECCV 2016, Springer International Publishing. Lecture Notes in Computer Science.10.1007/978-3-319-46454-1 ↗
18. Tan, M., Pang, R., and Le, Q.V. (2020). EfficientDet: Scalable and Efficient Object Detection. arXiv.10.1109/CVPR42600.2020.01079 ↗
19. Analysis of a Pulse Rate Variability Measurement Using a Smartphone Camera
20. Yadav, P., Choubisa, M., Shukla, A., and Radhika, R. (2024, January 9–10). Cardiac Pulse Monitoring Through Multi-Scale Spectrum. Proceedings of the 2024 International Conference on Advances in Modern Age Technologies for Health and Engineering Science (AMATHE), Shivamogga, India.10.1109/AMATHE61652.2024.10582227 ↗
21. Accuracy of oscillometric-based blood pressure monitoring devices: Impact of pulse volume, arrhythmia, respiratory artifact10.1038/s41371-023-00856-4 ↗
22. Phase-based video motion processing10.1145/2461912.2461966 ↗
23. Wadhwa, N., Rubinstein, M., Durand, F., and Freeman, W.T. (2014, January 2–4). Riesz pyramids for fast phase-based video magnification. Proceedings of the 2014 IEEE International Conference on Computational Photography (ICCP), Santa Clara, CA, USA.10.1109/ICCPHOT.2014.6831820 ↗
24. Elgharib, M., Hefeeda, M., Durand, F., and Freeman, W.T. (2015, January 7–12). Video magnification in presence of large motions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA.10.1109/CVPR.2015.7299039 ↗
25. Oh, T.-H., Jaroensri, R., Kim, C., Elgharib, M., Durand, F.E., Freeman, W.T., and Matusik, W. (2018, January 8–14). Learning-Based Video Motion Magnification. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany. Available online: https://openaccess.thecvf.com/content_ECCV_2018/html/Tae-Hyun_Oh_Learning-based_Video_Motion_ECCV_2018_paper.html.10.1007/978-3-030-01225-0_39 ↗
26. Rahman, T., Krathu, W., and Arpnikanondt, C. (2023, January 21–24). Heart Rate Measurement on PC and Phone using Facial Videos. Proceedings of the 2023 15th International Conference on Knowledge and Smart Technology (KST), Phuket, Thailand.10.1109/KST57286.2023.10086729 ↗
27. Russell, S.J., and Norvig, P. (2023, October 10). Artificial Intelligence: A Modern Approach, Third Edition, Global Edition. Pearson. Available online: https://people.engr.tamu.edu/guni/csce421/files/AI_Russell_Norvig.pdf.
28. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C.L. (2015). Microsoft COCO: Common Objects in Context. arXiv.10.1007/978-3-319-10602-1_48 ↗
29. Box-Trainer Assessment System with Real-Time Multi-Class Detection and Tracking of Laparoscopic Instruments, using CNN10.12700/APH.19.2.2022.2.1 ↗
30. Yuan, L., Qu, Z., Zhao, Y., Zhang, H., and Nian, Q. (2017, January 25–26). A convolutional neural network based on TensorFlow for face recognition. Proceedings of the 2017 IEEE 2nd Advanced Information Technology, Electronic and Automation Control Conference (IAEAC), Chongqing China.10.1109/IAEAC.2017.8054070 ↗
31. Sukri, S.S., and Ruhaiyem, N.I.R. (2019). iFR: A New Framework for Real-Time Face Recognition with Deep Learning. International Visual Informatics Conference, Springer.10.1007/978-3-030-34032-2_26 ↗
32. A Convolutional Neural Network (CNN) Approach to Detect Face Using Tensorflow and Keras
33. Alabib-65: A realistic dataset for Algerian sign language recognition. ACM Trans10.1145/3596909 ↗
34. Detecting the Absence of Lung Sliding in Ultrasound Videos Using 3D Convolutional Neural Networks10.12700/APH.20.6.2023.6.3 ↗
35. Corrective Focus Detection in Italian Speech Using Neural Networks
36. Bae, G., de La Gorce, M., Baltrušaitis, T., Hewitt, C., Chen, D., Valentin, J., Cipolla, R., and Shen, J. (2024, January 3–8). DigiFace-1M: 1 Million Digital Face Images for Face Recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA. Available online: https://openaccess.thecvf.com/content/WACV2023/html/Bae_DigiFace-1M_1_Million_Digital_Face_Images_for_Face_Recognition_WACV_2023_paper.html.10.1109/WACV56688.2023.00352 ↗
37. Liu, Z., Luo, P., Wang, X., and Tang, X. (2015, January 7–13). Deep Learning Face Attributes in the Wild. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile. Available online: https://openaccess.thecvf.com/content_iccv_2015/html/Liu_Deep_Learning_Face_ICCV_2015_paper.html.10.1109/ICCV.2015.425 ↗
38. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., and Zisserman, A. (2018, January 15–19). VGGFace2: A Dataset for Recognising Faces across Pose and Age. Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), Xi’an, China.10.1109/FG.2018.00020 ↗
39. Bansal, A., Nanduri, A., Castillo, C.D., Ranjan, R., and Chellappa, R. (2017, January 1–4). UMDFaces: An annotated face dataset for training deep networks. Proceedings of the 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, CO, USA.10.1109/BTAS.2017.8272731 ↗
40. Beveridge, J.R., Bowyer, K.W., Flynn, P.J., Cheng, S., Phillips, P.J., Bolme, D.S., Draper, B.A., Givens, G.H., Lui, Y.M., and Teli, M.N. (October, January 29). The challenge of face recognition from digital point-and-shoot cameras. Proceedings of the 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), Arlington, VA, USA.10.1109/BTAS.2013.6712704 ↗
41. Yang, B., Yan, J., Lei, Z., and Li, S.Z. (2015, January 4–8). Fine-grained evaluation on face detection in the wild. Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia.
42. Wu, B., Iandola, F., Jin, P.H., and SqueezeDet, K.K. (2017, January 21–26). Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, HI, USA. Available online: https://openaccess.thecvf.com/content_cvpr_2017_workshops/w4/html/Wu_SqueezeDet_Unified_Small_CVPR_2017_paper.html.10.1109/CVPRW.2017.60 ↗
43. Fu, C.-Y., Liu, W., Ranga, A., Tyagi, A., and Berg, A.C. (2017). DSSD: Deconvolutional Single Shot Detector. arXiv.
44. Jeong, J., Park, H., and Kwak, N. (2017). Enhancement of SSD by concatenating feature maps for object detection. arXiv.10.5244/C.31.76 ↗
45. An enhanced SSD with feature fusion and visual reasoning for object detection10.1007/s00521-018-3486-1 ↗
46. Cao, G., Xie, X., Yang, W., Liao, Q., Shi, G., and Wu, J. (2017, January 14–16). Feature-fused SSD: Fast detection for small objects. Proceedings of the Ninth International Conference on Graphic and Image Processing (ICGIP 2017), International Society for Optics and Photonics, Qingdao, China.
47. Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., and Guadarrama, S. (2017, January 21–26). Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA. Available online: https://openaccess.thecvf.com/content_cvpr_2017/html/Huang_SpeedAccuracy_Trade-Offs_for_CVPR_2017_paper.html.10.1109/CVPR.2017.351 ↗
48. Kwaśniewska, A., Rumiński, J., and Rad, P. (2017, January 17–19). Deep features class activation map for thermal face detection and tracking. Proceedings of the 2017 10th International Conference on Human System Interactions (HSI), Ulsan, Republic of Korea.10.1109/HSI.2017.8004993 ↗
49. Caramihale, T., Popescu, D., and Ichim, L. (2018). Emotion classification using a tensorflow generative adversarial network implementation. Symmetry, 10.10.3390/sym10090414 ↗
50. Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow10.1007/s00371-019-01628-3 ↗
51. Face detection techniques: A review10.1007/s10462-018-9650-2 ↗
52. Enhanced Multiview attention network with random interpolation resize for few-shot surface defect detection10.1007/s00530-024-01643-y ↗
53. Alam, S., Singh, S.P., and Abeyratne, U. (2017, January 11–15). Considerations of handheld respiratory rate estimation via a stabilized Video Magnification approach. Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Jeju Island, Republic of Korea.10.1109/EMBC.2017.8037805 ↗
54. Video-based HR measurement using adaptive facial regions with multiple color spaces10.1016/j.bbe.2023.12.001 ↗