journal article·2024-08-16Open access

Camera-Sourced Heart Rate Synchronicity: A Measure of Immersion in Audiovisual Experiences

Joseph Williams, Jon Francombe, Damian Murphy

Applied Sciences· Volume 14· Issue 16

Abstract

Audio presentation is often attributed as being capable of influencing a viewer’s feeling of immersion during an audiovisual experience. However, there is limited empirical research supporting this claim. This study aimed to explore this effect by presenting a clip renowned for its immersive soundtrack to two groups of participants with either high-end or basic audio presentation. To measure immersion, a novel method is applied, which utilises a camera instead of an electroencephalogram (ECG) for acquiring a heart rate synchronisation feature. The results of the study showed no difference in the feature, or in the responses to an established immersion questionnaire, between the two groups of participants. However, the camera-sourced HR synchronicity feature correlated with the results of the immersion questionnaire. Moreover, the camera-sourced HR synchronicity feature was found to correlate with an equivalent feature sourced from synchronously recorded ECG data. Hence, this shows the viability of using a camera instead of an ECG sensor to quantify heart rate synchronisation but suggests that audio presentation alone is not capable of eliciting a measurable difference in the feeling of immersion in this context.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Color perception and design · Multisensory perception and integration

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

AudioLab, School of Physics Engineering and Technology, University of York, Genesis 6, Heslington, York YO10 5DQ, UK

Bang & Olufsen a/s, 7600 Struer, Denmark

Funding

UK Arts and Humanities Research Council (AHRC) XR Stories Creative Industries Cluster project · University of York-funded PhD studentship · Bang & Olufsen

References (82)

Show source-provided references
1. Cuarón, A. (2024, April 29). Gravity. Motion Picture byWarner Brothers Pictures, 2013. Available online: https://www.imdb.com/title/tt1454468/.
2. Fernando, I.Z.R., Simon, v.d.B., Loépez, M., and Pauletto, S. (2021). Immersive Continuity: Long Takes, 3-D Sound, and the Impression of Reality in the Cinema of Alfonso Cuaroén. [Ph.D. Thesis, University of York].
3. Sound, space, gravity: A kaleidoscopic hearing (part I)10.3366/sound.2016.0079 ↗
4. Sound, space, gravity: A kaleidoscopic hearing (part II)10.3366/sound.2016.0091 ↗
5. Mera, M. (2016). Towards 3-D sound: Spatial presence and the space vacuum. The Palgrave Handbook of Sound Design and Music in Screen Media, Palgrave Macmillan.10.1057/978-1-137-51680-0_7 ↗
6. Cuarón, A. (2024, April 29). Gravity: Director Alfonso Cuarón on Dolby Atmos. Available online: https://www.youtube.com/watch?v=c7dKt80o0qE.
7. (2024, April 29). Dolby Atmos—Official Site. Available online: https://www.dolby.com/technologies/dolby-atmos/.
8. (2024, April 29). Gravity Blu-Ray (Special Edition). Available online: https://www.blu-ray.com/movies/Gravity-Blu-ray/138954/.
9. Hammond, H. (2023). Developing Continuous Measures of Audience Immersion. [Ph.D. Thesis, University of Bristol].
10. Audience immersion: Validating attentional and physiological measures against self-report10.1186/s41235-023-00475-0 ↗
11. Cognitive processing of a common stimulus synchronizes brains, hearts, and eyes10.1093/pnasnexus/pgac020 ↗
12. Conscious processing of narrative stimuli synchronizes heart rate between individuals
13. Gibbs, H.J., Czepiel, A., and Egermann, H. (2023). Physiological synchrony and shared flow state in Javanese gamelan: Positively associated while improvising, but not for traditional performance. Front. Psychol., 14.10.3389/fpsyg.2023.1214505 ↗
14. Mark Andrews, B.C. (2024, April 29). Brave. Motion Picture by Walt Disney Pictures and Pixar Animation Studios. Available online: https://www.imdb.com/title/tt1217209/.
15. Dolby Laboratories (2024, April 29). Speaker Setup Guides. Available online: https://www.dolby.com/about/support/guide/speaker-setup-guides/.
16. Williams, J., Shepstone, S., and Murphy, D. (2022, January 15–17). Understanding Immersion in the Context of Films with Spatial Audio. Proceedings of the Audio Engineering Society Conference: 2022 AES International Conference on Audio for Virtual and Augmented Reality, Redmond, VA, USA.
17. Bech, S., and Zacharov, N. (2006). Perceptual Audio Evaluation—Theory, Method and Application, John Wiley & Sons.10.1002/9780470869253 ↗
18. Agrawal, S.R., and Bech, S. (2022). Immersion in audiovisual experiences. Sonic Interactions in Virtual Environments, Springer International Publishing.10.1007/978-3-031-04021-4_11 ↗
19. Williams, J., Francombe, J., and Murphy, D. (2023, January 23–25). Exploring the influence of multichannel soundtracks on film immersion. Proceedings of the Audio Engineering Society Conference: AES 2023 International Conference on Spatial and Immersive Audio, Huddersfield, UK.
20. Defining immersion: Literature review and implications for research on audiovisual experiences10.17743/jaes.2020.0039 ↗
21. Murray, J.H. (1998). Hamlet on the Holodeck: The Future of Narrative in Cyberspace, The MIT Press.
22. Rigby, J.M., Brumby, D.P., Gould, S.J.J., and Cox, A.L. (2019, January 5–7). Development of a questionnaire to measure immersion in video media: The Film IEQ. Proceedings of the 2019 ACM International Conference on Interactive Experiences for TV and Online Video, Salford, UK. TVX ’19.10.1145/3317697.3323361 ↗
23. Method for subjective assessment of immersion in audiovisual experiences10.17743/jaes.2021.0013 ↗
24. Influence of Changes in Audio Spatialization on Immersion in Audiovisual Experiences10.17743/jaes.2022.0034 ↗
25. Initial validation and reliability of the single-item measure of immersion
26. Immersion Measurement in Watching Videos Using Eye-tracking Data10.1109/TAFFC.2022.3209311 ↗
27. The Why, What, and How of Immersive Experience10.1109/ACCESS.2020.2993646 ↗
28. Lessiter, J., Freeman, J., and Davidoff, J. (2001, January 21–23). Really hear? The effects of audio quality on presence. Proceedings of the Fourth Annual International Workshop on Presence, Philadelphia, PA, USA.
29. Effects of down-mix algorithms on quality of surround sound
30. Effects of Bandwidth Limitation on Audio Quality in Consumer Multichannel Audiovisual Delivery Systems
31. Lipscomb, S.D., and Kerins, M. (2004, January 3–7). An empirical investigation into the effect of presentation mode in the cinematic and music listening experience. Proceedings of the 8th International Conference on Music Perception & Cognition, Evanston, IL, USA.
32. Oramus, T., and Neubauer, P. (2019, January 16–19). Comparison Study of Listeners’ Perception of 5.1 and Dolby Atmos. Proceedings of the Audio Engineering Society Convention 147, New York, NY, USA.
33. Langiulli, N., Calbi, M., Sbravatti, V., Umiltà, M.A., and Gallese, V. (2023). The effect of Surround sound on embodiment and sense of presence in cinematic experience: A behavioral and HD-EEG study. Front. Neurosci., 17.10.3389/fnins.2023.1222472 ↗
34. Kerins, M. (2013). Understanding the impact of surround sound in multimedia. The Psychology of Music in Multimedia, Oxford University Press.10.1093/acprof:oso/9780199608157.003.0016 ↗
35. Hedges, J., Sazdov, R., and Johnston, A. (2023, January 5–7). Measuring the influence of audio on immersive experience in extended reality and digital games: A systematic review. Proceedings of the 2023 Immersive and 3D Audio: From Architecture to Automotive (I3DA), Bologna, Italy.10.1109/I3DA57090.2023.10289267 ↗
36. Potter, T., Cvetković, Z., and De Sena, E. (2022). On the relative importance of visual and spatial audio rendering on vr immersion. Front. Signal Process., 2.10.3389/frsip.2022.904866 ↗
37. In the dark cube: Movie theater context enhances the valuation and aesthetic experience of watching films10.1037/aca0000295 ↗
38. The effects of screen size and message content on attention and arousal10.1207/s1532785xmep0101_4 ↗
39. Review on Psychological Stress Detection Using Biosignals10.1109/TAFFC.2019.2927337 ↗
40. Lahnakoski, J., and Change, L. (2024, April 29). Intersubject Correlation—Naturalistic Data Analysis. Available online: https://naturalistic-data.org/content/Intersubject_Correlation.html.
41. Measuring shared responses across subjects using intersubject correlation
42. Laerd Statistics (2024, April 29). Pearson Correlation Coefficient Statistical Guide. Available online: https://statistics.laerd.com/statistical-guides/pearson-correlation-coefficient-statistical-guide.php.
43. Laerd Statistics (2024, April 29). Independent t-Test Using SPSS Statistics. Available online: https://statistics.laerd.com/spss-tutorials/independent-t-test-using-spss-statistics.php.
44. Morales-Fajardo, H.M., Rodríguez-Arce, J., Gutiérrez-Cedeño, A., Viñas, J.C., Reyes-Lagos, J.J., Abarca-Castro, E.A., Ledesma-Ramírez, C.I., and Vilchis-González, A.H. (2022). Towards a non-contact method for identifying stress using remote photoplethysmography in academic environments. Sensors, 22.10.3390/s22103780 ↗
45. Sabour, R.M., Benezeth, Y., De Oliveira, P., Chappe, J., and Yang, F. (2021). Ubfc-phys: A multimodal database for psychophysiological studies of social stress. IEEE Trans. Affect. Comput.
46. Zhou, K., Schinle, M., Weimar, S., Gerdes, M., Stock, S., and Stork, W. (2022, January 6–8). End-to-End Deep Learning for Stress Recognition Using Remote Photoplethysmography. Proceedings of the 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Las Vegas, NV, USA.10.1109/BIBM55620.2022.9995577 ↗
47. pyVHR: A Python framework for remote photoplethysmography10.7717/peerj-cs.929 ↗
48. Mitsubishi Electric (2024, April 29). Mitsubishi LDT422V LCD Display. Available online: https://www.manuals.co.uk/mitsubishi/ldt422v/manual.
49. Genelec (2024, April 29). Genelec 8010A Studio Monitor. Available online: https://www.genelec.com/8010a.
50. Genelec (2024, April 29). Genelec 8030C Studio Monitor. Available online: https://www.genelec.com/8030c.
51. Genelec (2024, April 29). Genelec 7040A Studio Subwoofer. Available online: https://www.genelec.com/7040a.
52. Genelec (2024, April 29). Genelec 8040A Studio Monitor. Available online: https://www.genelec.com/previous-models/8040a.
53. Mulcahy, J. (2023, November 06). REW Room Acoustics and Audio Device Measurement and Analysis Software. Available online: https://www.roomeqwizard.com/.
54. Earthworks Audio (2024, April 29). M30 Measurement Microphone. Available online: https://earthworksaudio.com/wp-content/uploads/2018/07/M30-Data-Sheet-2018.pdf.
55. (2023, November 06). Dolby Reference Player. Available online: https://professional.dolby.com/product/.
56. (2024, April 30). NTi Audio XL2 Audio and Acoustic Analyzer. Available online: https://www.nti-audio.com/en/products/xl2-sound-level-meter.
57. International Telecommunication Union (2024, April 29). Recommendation ITU-R BS.775-4 Multi-Channel Stereophonic sound System with and without Accompanying Picture. Available online: https://www.itu.int/rec/R-REC-BS.775-4-202212-I/en.
58. (2024, April 30). Tenma 72-860A Loudness Meter. Available online: https://www.tenma.com.
59. (2024, April 30). Neumann KU 100 Dummy Head Microphone. Available online: https://www.neumann.com/homestudio/en/ku-100.
60. Canon Inc. (2024, April 29). Canon EOS 700D. Available online: https://www.canon.co.uk/support/consumer/products/cameras/eos/eos-700d.html.
61. McDuff, D.J., Blackford, E.B., and Estepp, J.R. (June, January 30). The impact of video compression on remote cardiac pulse measurement using imaging photoplethysmography. Proceedings of the 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA.10.1109/FG.2017.17 ↗
62. Effects of frame rate and image resolution on pulse rate measured using multiple camera imaging photoplethysmography
63. Softonic (2023, June 26). LosslessCut GitHub. Available online: https://github.com/mifi/lossless-cut.
64. Harvey, P. (2024, April 29). Exiftool. Available online: https://exiftool.org/.
65. Polar Electro (2024, April 29). Polar H10 Heart Rate Sensor User Manual, Available online: https://support.polar.com/e_manuals/h10-heart-rate-sensor/polar-h10-user-manual-english/manual.pdf.
66. Jukka Happonen (2024, April 29). Polar Sensor Logger. Available online: https://play.google.com/store/apps/details?id=com.j_ware.polarsensorlogger.
67. AudioLab (2024, April 29). Github: Camera-Sourced-Heart-Rate-Synchronicity, Nov. Available online: https://github.com/AudioLabYork/Camera-Sourced-Heart-Rate-Synchronicity.
68. Williams, J., Francombe, J., and Murphy, D. (2023). Evaluating the Influence of Room Illumination on Camera-Based Physiological Measurements for the Assessment of Screen-Based Media. Appl. Sci., 13.10.3390/app13148482 ↗
69. Rooney, B., Hennessy, E., and Bálint, K. (2014, January 11–14). Viewer versus film: Exploring interaction effects of immersion and cognitive stance on the heart rate and self-reported engagement of viewers of short films. Proceedings of the Poster presentation at the Society for Cognitive Studies of the Moving Image, Franklin & Marshall College, Lancaster, PA, USA.
70. The permutation testing approach: A review
71. Laerd Statistics (2024, April 29). Spearman’s Rank-Order Correlation Using SPSS Statistics. Available online: https://statistics.laerd.com/spss-tutorials/spearmans-rank-order-correlation-using-spss-statistics.php.
72. Monk, E. (2023, June 26). Monk Scale. Available online: https://skintone.google/the-scale.
73. Statista (2024, April 29). Favorite Movie Genres in the U.S. by Age. Available online: https://www.statista.com/statistics/949810/favorite-movie-genres-in-the-us-by-age/.
74. Nowara, E.M., McDuff, D., and Veeraraghavan, A. (2020, January 14–19). A meta-analysis of the impact of skin tone and gender on non-contact photoplethysmography measurements. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Seattle, WA, USA.10.1109/CVPRW50498.2020.00150 ↗
75. Cheong, J.H., Molani, Z., Sadhukha, S., and Chang, L.J. (2023). Synchronized affect in shared experiences strengthens social connection. Commun. Biol., 6.10.1038/s42003-023-05461-2 ↗
76. Bühler, B., Bozkir, E., Deininger, H., Gerjets, P., Trautwein, U., and Kasneci, E. (2024). On Task and in Sync: Examining the Relationship between Gaze Synchrony and Self-Reported Attention During Video Lecture Learning. arXiv.10.1145/3655604 ↗
77. A psychometric perspective on psychophysiological measures10.1037/1040-3590.7.3.387 ↗
78. Kerins, M. (2010). Beyond Dolby (Stereo): Cinema in the Digital Sound Age, Indiana University Press.
79. When our hearts beat together: Cardiac synchrony as an entry point to understand dyadic co-regulation in couples10.1111/psyp.13739 ↗
80. Phillips, T. (2024, April 29). Joker. Motion Picture by Warner Bros. Available online: https://www.imdb.com/title/tt7286456/.
81. Hearing reality in Joker10.1080/17400309.2020.1861883 ↗
82. Bolen, K. (2021). Mixing 3D audio for film. 3D Audio, Routledge.10.4324/9780429491214-14 ↗