journal article·2024-07-10Open access

A Review of Recent Literature on Audio-Based Pseudo-Haptics

Sandy Abdo, Bill Kapralos, KC Collins, Adam Dubrowski

Applied Sciences· Volume 14· Issue 14

Abstract

Immersive virtual learning environments (iVLEs), particularly serious games and virtual simulations, typically ignore psychomotor skills development, partly due to the difficulty and cost associated with accurately replicating touch. Simulating touch, also known as haptics, requires specialized haptic devices that are not widely accessible at the consumer-level. Using visual (and/or auditory) cues, pseudo-haptics aims to mimic touch sensations without haptic devices. Although pseudo-haptics has predominantly focused on visual cues, a 2019 review by Collins and Kapralos on pseudo-haptics emphasized the role of auditory cues and cross-modal interactions. Since then, great advancements, notably during the COVID-19 pandemic’s shift to remote learning, have been made. Here, we build upon the work of Collins and Kapralos with a narrative review on audio-based pseudo-haptics. This narrative review explores 17 articles obtained from the Google Scholar, RefSeek, Scopus, and PubMed databases, with the aim of providing a comprehensive summary of the progress in this field since 2019. Pseudo-haptics presents a viable alternative to simulate various aspects of touch, including weight, stiffness, roughness, dampness, force, and glossiness, when haptic devices are unavailable, enhancing immersion and providing the potential to improve psychomotor skill training within iVLEs.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Tactile and Sensory Interactions · Multisensory perception and integration · Visual perception and processing mechanisms

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

maxSIMhealth Group, Ontario Tech University, Oshawa, ON L1G 0C5, Canada

School of Information Technology, Carleton University, Ottawa, ON K1S 5B6, Canada

Funding

Natural Sciences and Engineering Research Council of Canada (NSERC)

References (81)

Show source-provided references
1. Baek, Y., Ko, R., and Marsh, T. (2014). The Future of Serious Games and Immersive Technologies and Their Impact on Society. Trends and Applications of Serious Gaming and Social Media, Springer. Gaming Media and Social Effects.10.1007/978-981-4560-26-9 ↗
2. The State of Immersive Technology Research: A Literature Analysis10.1016/j.chb.2018.04.019 ↗
3. Hall, S., and Takahashi, R. (2017, January 17–20). Augmented and Virtual Reality: The Promise and Peril of Immersive Technologies. Proceedings of the World Economic Forum, Davos-Klosters, Switzerland.
4. Lessons Learned in Preparing for and Responding to the Early Stages of the COVID-19 Pandemic: One Simulation’s Program Experience Adapting to the New Normal10.1186/s41077-020-00128-y ↗
5. (2024, June 24). Straits Research E-Learning Market to Reach USD 645 Billion in Market Size by 2030, Growing at a CAGR of 13%: Straits Research. Available online: https://www.infoprolearning.com/blog/elearning-localization-your-secret-code-to-reach-the-global-audience/.
6. (2024, June 24). Zion Research Serious Games Market Size, Share, Growth Report 2030. Available online: https://www.zionmarketresearch.com/report/serious-games-market.
7. Mather, G. (2016). Foundations of Sensation and Perception, Psychology Press.10.4324/9781315672236 ↗
8. Dubois, D., Cance, C., Coler, M., Paté, A., and Guastavino, C. (2021). Sensory Experiences: Exploring Meaning and the Senses, John Benjamins Publishing Company.10.1075/celcr.24 ↗
9. Spielman, M.R., Dumper, K., Jenkins, W., Lacombe, A., Lovett, M.D., and Perlmutter, M. (2021). Psychology, BCcampus. [H5P ed.].
10. Wilson, K.A. (2021). How Many Senses? Multisensory Perception Beyond the Five Senses. Sabah Ülkesi, IGMG. Available online: https://philpapers.org/rec/WILMP-8.
11. Bradley, M., and Butler, S. (2018). Touch and the Ancient Senses, Routledge. The Senses in Antiquity.
12. Hutmacher, F. (2019). Why Is There So Much More Research on Vision Than on Any Other Sensory Modality?. Front. Psychol., 10.10.3389/fpsyg.2019.02246 ↗
13. Evaluating Whether Sight Is the Most Valued Sense10.1001/jamaophthalmol.2019.3537 ↗
14. Classen, C. (2012). The Deepest Sense: A Cultural History of Touch, University of Illinois Press.10.5406/illinois/9780252034930.001.0001 ↗
15. A Review of Haptic Feedback in Tele-Operated Robotic Surgery10.1080/03091902.2020.1772391 ↗
16. See, A.R., Choco, J.A.G., and Chandramohan, K. (2022). Touch, Texture and Haptic Feedback: A Review on How We Feel the World around Us. Appl. Sci., 12.10.3390/app12094686 ↗
17. Gallace, A., and Spence, C. (2014). In Touch with the Future: The Sense of Touch from Cognitive Neuroscience to Virtual Reality, Oxford University Press.10.1093/acprof:oso/9780199644469.001.0001 ↗
18. Haptic Devices: Wearability-Based Taxonomy and Literature Review10.1109/ACCESS.2022.3202986 ↗
19. Siciliano, B., and Khatib, O. (2016). Haptics. Springer Handbook of Robotics, Springer International Publishing.10.1007/978-3-319-32552-1 ↗
20. Oakley, I., McGee, M.R., Brewster, S., and Gray, P. (2000, January 1–6). Putting the Feel in ‘look and Feel’. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Hague, The Netherlands.10.1145/332040.332467 ↗
21. Cutaneous Feedback of Fingertip Deformation and Vibration for Palpation in Robotic Surgery10.1109/TBME.2015.2455932 ↗
22. Recent Advances in Multi-Mode Haptic Feedback Technologies towards Wearable Interfaces10.1016/j.mtphys.2021.100602 ↗
23. Lederman, S., and Klatzky, R.L. (2009). Human Haptics. Encyclopedia of Neuroscience, Academic Press.10.1016/B978-008045046-9.01905-7 ↗
24. Johansson, R.S., and Flanagan, J.R. (2009). Sensory Control of Object Manipulation. Sensorimotor Control of Grasping: Physiology and Pathophysiology, Cambridge University Press.10.1017/CBO9780511581267.012 ↗
25. Klatzky, R., and Lederman, S. (2003). The Haptic Identification of Everyday Life Objects. Touching for Knowing: Cognitive Psychology of Haptic Manual Perception, John Benjamins Publishing Company.10.1075/aicr.53.11kla ↗
26. Sensory Signals in Neural Populations Underlying Tactile Perception and Manipulation10.1146/annurev.neuro.26.041002.131032 ↗
27. Upper Limb Asymmetries in the Matching of Proprioceptive Versus Visual Targets10.1152/jn.90259.2008 ↗
28. Haptic Perception: A Tutorial10.3758/APP.71.7.1439 ↗
29. Haptics: The Present and Future of Artificial Touch Sensation10.1146/annurev-control-060117-105043 ↗
30. Haptics to Improve Task Performance in People with Disabilities: A Review of Previous Studies and a Guide to Future Research with Children with Disabilities
31. Panariello, D., Caporaso, T., Grazioso, S., Di Gironimo, G., Lanzotti, A., Knopp, S., Pelliccia, L., Lorenz, M., and Klimant, P. (2019, January 25–27). Using the KUKA LBR Iiwa Robot as Haptic Device for Virtual Reality Training of Hip Replacement Surgery. Proceedings of the 2019 Third IEEE International Conference on Robotic Computing (IRC), Naples, Italy.10.1109/IRC.2019.00094 ↗
32. Feel It on Your Fingers: Dataglove with Vibrotactile Feedback for Virtual Reality and Telerobotics10.1007/978-3-030-23807-0_31 ↗
33. Pseudo-Haptics: Leveraging Cross-Modal Perception in Virtual Environments10.1080/17458927.2019.1619318 ↗
34. D’Abbraccio, J., Massari, L., Prasanna, S., Baldini, L., Sorgini, F., Airò Farulla, G., Bulletti, A., Mazzoni, M., Capineri, L., and Menciassi, A. (2019). Haptic Glove and Platform with Gestural Control for Neuromorphic Tactile Sensory Feedback in Medical Telepresence. Sensors, 19.10.3390/s19030641 ↗
35. Aldridge, R.J., Carr, K., England, R., Meech, J.F., and Solomonides, T. Getting a Grasp on Virtual Reality. Proceedings of the Conference Companion on Human Factors in Computing Systems Common Ground—CHI ’96.
36. User Representations in Human-Computer Interaction10.1080/07370024.2020.1724790 ↗
37. Elastic Images: Perceiving Local Elasticity of Images through a Novel Pseudo-Haptic Deformation Effect10.1145/2501599 ↗
38. Pseudo-Haptic Display of Mass and Mass Distribution During Object Rotation in Virtual Reality10.1109/TVCG.2020.2973056 ↗
39. A Study of the Modification of the Speed and Size of the Cursor for Simulating Pseudo-Haptic Bumps and Holes10.1145/1402236.1402238 ↗
40. Survey of Pseudo-Haptics: Haptic Feedback Design and Application Proposals10.1109/TOH.2021.3077619 ↗
41. Lécuyer, A., Coquillart, S., Kheddar, A., Richard, P., and Coiffet, P. (2000, January 18–22). Pseudo-Haptic Feedback: Can Isometric Input Devices Simulate Force Feedback?. Proceedings of the IEEE Virtual Reality 2000 (Cat. No.00CB37048), New Brunswick, NJ, USA.
42. A Pseudo-Haptic Method Using Auditory Feedback: The Role of Delay, Frequency, and Loudness of Auditory Feedback in Response to a User’s Button Click in Causing a Sensation of Heaviness10.1109/ACCESS.2022.3172324 ↗
43. Garlinska, M., Osial, M., Proniewska, K., and Pregowska, A. (2023). The Influence of Emerging Technologies on Distance Education. Electronics, 12.10.3390/electronics12071550 ↗
44. Simulating Haptic Feedback Using Vision: A Survey of Research and Applications of Pseudo-Haptic Feedback10.1162/pres.18.1.39 ↗
45. Users’ Guides to the Medical Literature: VI. How to Use an Overview10.1001/jama.1994.03520170077040 ↗
46. A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies10.1111/j.1471-1842.2009.00848.x ↗
47. Writing Narrative Literature Reviews for Peer-Reviewed Journals: Secrets of the Trade10.1016/S0899-3467(07)60142-6 ↗
48. PRISMA2020: An R Package and Shiny App for Producing PRISMA 2020-compliant Flow Diagrams, with Interactivity for Optimised Digital Transparency and Open Synthesis10.1002/cl2.1230 ↗
49. Wouters, P., and Van Oostendorp, H. (2017). Fidelity and Multimodal Interactions. Instructional Techniques to Facilitate Learning and Motivation of Serious Games, Springer International Publishing.10.1007/978-3-319-39298-1 ↗
50. Melaisi, M., Rojas, D., Kapralos, B., Uribe-Quevedo, A., and Collins, K. (2018). Multimodal Interaction of Contextual and Non-Contextual Sound and Haptics in Virtual Simulations. Informatics, 5.10.3390/informatics5040043 ↗
51. Towards Multisensory Perception: Modeling and Rendering Sounds of Tool-Surface Interactions10.1109/TOH.2020.2966192 ↗
52. Chan, S., Tymms, C., and Colonnese, N. (2021, January 6–9). Hasti: Haptic and Audio Synthesis for Texture Interactions. Proceedings of the 2021 IEEE World Haptics Conference (WHC), Montreal, QC, Canada.10.1109/WHC49131.2021.9517177 ↗
53. An Alternative Method of Audio-Tactile Presentation of Graphical Information in Mathematics Adapted to the Needs of Blind10.1016/j.ijhcs.2023.103122 ↗
54. Creating Pseudo-Tactile Feedback in Virtual Reality Using Shared Crossmodal Properties of Audio and Tactile Feedback
55. Research on a Method of Conveying Material Sensations through Sound Effects10.1080/09298215.2023.2187311 ↗
56. Crossmodal Perception in Virtual Reality10.1007/s11042-019-7331-z ↗
57. Understanding Virtual Drilling Perception Using Sound, and Kinesthetic Cues Obtained with a Mouse and Keyboard10.1007/s12193-023-00412-x ↗
58. Speicher, M., Ehrlich, J., Gentile, V., Degraen, D., Sorce, S., and Krüger, A. (2019, January 4–9). Pseudo-Haptic Controls for Mid-Air Finger-Based Menu Interaction. Proceedings of the Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290607.3312927 ↗
59. Psychophysical Effects of Experiencing Burning Hands in Augmented Reality10.1007/978-3-030-62655-6_5 ↗
60. Comparison of Three Feedback Modalities for Haptics Sensation in Remote Machine Manipulation10.1109/LRA.2021.3070301 ↗
61. Effects of Visual and Auditory Cues on Haptic Illusions for Active and Passive Touches in Mixed Reality10.1016/j.ijhcs.2021.102613 ↗
62. Puértolas Bálint, L.A., Althoefer, K., and Perez Macias, L.H. (2021, January 6–9). Virtual Reality Percussion Simulator for Medical Student Training. Proceedings of the 2021 IEEE 6th International Forum on Research and Technology for Society and Industry (RTSI), Naples, Italy.10.1109/RTSI50628.2021.9597234 ↗
63. Desnoyers-Stewart, J., Stepanova, E.R., Liu, P., Kitson, A., Pennefather, P.P., Ryzhov, V., and Riecke, B.E. (2023, January 23–28). Embodied Telepresent Connection (ETC): Exploring Virtual Social Touch Through Pseudohaptics. Proceedings of the Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, Hamburg, Germany.10.1145/3544549.3585843 ↗
64. Kurzweg, M., Letter, M., and Wolf, K. (2023, January 3–6). Vibrollusion: Creating a Vibrotactile Illusion Induced by Audiovisual Touch Feedback. Proceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia, Vienna, Austria.10.1145/3626705.3627790 ↗
65. Pseudo-Haptic Feedback Design for Virtual Activities in Human Computer Interface10.1007/978-3-031-35634-6_18 ↗
66. Lécuyer, A., Burkhardt, J.-M., and Etienne, L. (2004, January 24–29). Feeling Bumps and Holes without a Haptic Interface: The Perception of Pseudo-Haptic Textures. Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vienna, Austria.10.1145/985692.985723 ↗
67. Moosavi, M.S., Raimbaud, P., Guillet, C., Plouzeau, J., and Merienne, F. (2023). Weight Perception Analysis Using Pseudo-Haptic Feedback Based on Physical Work Evaluation. Front. Virtual Real., 4.10.3389/frvir.2023.973083 ↗
68. Mori, S., Kataoka, Y., and Hashiguchi, S. (2022, January 12–16). Exploring Pseudo-Weight in Augmented Reality Extended Displays. Proceedings of the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Christchurch, New Zealand.10.1109/VR51125.2022.00091 ↗
69. Costes, A., Argelaguet, F., Danieau, F., Guillotel, P., and Lécuyer, A. (2019). Touchy: A Visual Approach for Simulating Haptic Effects on Touchscreens. Front. ICT, 6.10.3389/fict.2019.00001 ↗
70. Crandall, R., and Karadoğan, E. (2021). Designing Pedagogically Effective Haptic Systems for Learning: A Review. Appl. Sci., 11.10.3390/app11146245 ↗
71. A Survey on Haptic Technologies for Mobile Augmented Reality10.1145/3465396 ↗
72. Bouzbib, E., Bailly, G., Haliyo, S., and Frey, P. (2021, January 13). “Can I Touch This?”: Survey of Virtual Reality Interactions via Haptic Solutions: Revue de Littérature Des Interactions En Réalité Virtuelle Par Le Biais de Solutions Haptiques. Proceedings of the 32e Conférence Francophone sur l’Interaction Homme-Machine, Virtual Event.10.1145/3450522.3451323 ↗
73. Kern, T.A., Hatzfeld, C., and Abbasimoshaei, A. (2023). Haptics as an Interaction Modality. Engineering Haptic Devices, Springer International Publishing.10.1007/978-3-031-04536-3 ↗
74. A Systematic Review of Weight Perception in Virtual Reality: Techniques, Challenges, and Road Ahead10.1109/ACCESS.2021.3131525 ↗
75. Pusch, A., and Lécuyer, A. (2011, January 14–18). Pseudo-Haptics: From the Theoretical Foundations to Practical System Design Guidelines. Proceedings of the 13th International Conference on Multimodal Interfaces, Alicante, Spain.10.1145/2070481.2070494 ↗
76. Meyer, U., Becker, J., Draheim, S., and von Luck, K. (2021, January 8–13). Sensory Simulation in the Use of Haptic Proxies: Best Practices?. Proceedings of the Conference on Human Factors in Computing Systems, Online.
77. Double-Plate Fixation in Lateral Clavicle Fractures—A New Strategy
78. The Long-Term Outcome of Open Reduction and Internal Fixation of Stable Displaced Isolated Partial Articular Fractures of the Radial Head
79. Reproducibility and Reliability Analysis of the Luk Distal Radius and Ulna Classification for European Patients with Adolescent Idiopathic Scoliosis10.1302/1863-2548.15.200251 ↗
80. The Contributions of Kinesiology to Surgical Education10.2106/00004623-200412000-00029 ↗
81. Drilling Sounds Are Used by Surgeons and Intermediate Residents, but Not Novice Orthopedic Trainees, to Guide Drilling Motions