Multimodal Interaction Systems Based on Internet of Things and Augmented Reality: A Systematic Literature Review
Source abstract
Abstract
Technology developments have expanded the diversity of interaction modalities that can be used by an agent (either a human or machine) to interact with a computer system. This expansion has created the need for more natural and user-friendly interfaces in order to achieve effective user experience and usability. More than one modality can be provided to an agent for interaction with a system to accomplish this goal, which is referred to as a multimodal interaction (MI) system. The Internet of Things (IoT) and augmented reality (AR) are popular technologies that allow interaction systems to combine the real-world context of the agent and immersive AR content. However, although MI systems have been extensively studied, there are only several studies that reviewed MI systems that used IoT and AR. Therefore, this paper presents an in-depth review of studies that proposed various MI systems utilizing IoT and AR. A total of 23 studies were identified and analyzed through a rigorous systematic literature review protocol. The results of our analysis of MI system architectures, the relationship between system components, input/output interaction modalities, and open research challenges are presented and discussed to summarize the findings and identify future research and development avenues for researchers and MI developers.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Speech and dialogue systems · AI in Service Interactions · Social Robot Interaction and HRI
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology, 93187 Skellefteå, Sweden
Department of Digital Media, Ajou University, Suwon 16499, Korea
Funding
European Regional Development Fund · Ajou University
References (63)
Show source-provided references
1. Multimodal interfaces
2. A Review of Smart Homes—Past, Present, and Future10.1109/TSMCC.2012.2189204 ↗
3. Smart Cities: A Survey on Data Management, Security, and Enabling Technologies10.1109/COMST.2017.2736886 ↗
4. Networking and Communications in Autonomous Driving: A Survey10.1109/COMST.2018.2888904 ↗
5. Multimodal human–computer interaction: A survey10.1016/j.cviu.2006.10.019 ↗
6. Multimodal interaction: A review10.1016/j.patrec.2013.07.003 ↗
7. Internet of Things-IOT: Definition, Characteristics, Architecture, Enabling Technologies, Application & Future Challenges
8. Internet of Things (IoT): Opportunities, issues and challenges towards a smart and sustainable future10.1016/j.jclepro.2020.122877 ↗
9. Bhargava, M., Dhote, P., Srivastava, A., and Kumar, A. (2016, January 9–11). Speech enabled integrated AR-based multimodal language translation. Proceedings of the 2016 Conference on Advances in Signal Processing (CASP), Pune, India.10.1109/CASP.2016.7746170 ↗
10. Augmented Reality and Internet of Things – Implementation in Projects by Using Simplified Robotic Models10.18485/epmj.2018.8.2.4 ↗
11. Jo, D., and Kim, G.J. (2019). IoT + AR: Pervasive and augmented environments for “Digi-log” shopping experience. Hum.-Centric Comput. Inf. Sci., 9.10.1186/s13673-018-0162-5 ↗
12. Sun, Y., Armengol-Urpi, A., Reddy Kantareddy, S.N., Siegel, J., and Sarma, S. (2019, January 23–27). MagicHand: Interact with IoT Devices in Augmented Reality Environment. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.10.1109/VR.2019.8798053 ↗
13. Visualizing Toronto City Data with HoloLens: Using Augmented Reality for a City Model10.1109/MCE.2018.2797658 ↗
14. Architecture for Visualizing Indoor Air Quality Data with Augmented Reality Based Cognitive Internet of Things10.1007/978-3-030-44041-1_37 ↗
15. Mathews, N.S., Chimalakonda, S., and Jain, S. (2020). AiR—An Augmented Reality Application for Visualizing Air Pollution. arXiv.10.1109/VIS49827.2021.9623287 ↗
16. Augmented Reality in IoT10.1007/978-3-030-17642-6_13 ↗
17. Jo, D., and Kim, G.J. (2019). AR Enabled IoT for a Smart and Interactive Environment: A Survey and Future Directions. Sensors, 19.10.3390/s19194330 ↗
18. Investigating users’ intuitive interaction with complex artefacts10.1016/j.apergo.2009.04.010 ↗
19. Hogan, T., and Hornecker, E. (2016). Towards a Design Space for Multisensory Data Representation. Interact. Comput.10.1093/iwc/iww015 ↗
20. Liang, R., Liang, B., Wang, X., Zhang, T., Li, G., and Wang, K. (2016, January 1–3). A Review of Multimodal Interaction. Proceedings of the 2016 International Conference on Education, Management, Computer and Society, Shenyang, China.10.2991/emcs-16.2016.173 ↗
21. Badouch, A., Krit, S.D., Kabrane, M., and Karimi, K. (2018, January 19–21). Augmented Reality services implemented within Smart Cities, based on an Internet of Things Infrastructure, Concepts and Challenges: An overview. Proceedings of the Fourth International Conference on Engineering & MIS 2018—ICEMIS ’18, Istanbul, Turkey.10.1145/3234698.3234751 ↗
22. Al-Turjman, F. (2019). A Systematic Review of the Convergence of Augmented Reality, Intelligent Virtual Agents, and the Internet of Things. Artificial Intelligence in IoT, Springer International Publishing.
23. Picard, R.W. (2000). Affective Computing, 1st paperback ed., The MIT Press. OCLC: 247967780.
24. ISO (2018). Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts, ISO. ISO 9241-11:2018(en).
25. ISO (2013). Information Technology—Future Network—Problem Statement and Requirements—Part 6: Media Transport, ISO. ISO/IEC TR 29181-6:2013(en).
26. ITU (2017). P.10: Vocabulary for Performance, Quality of Service and Quality of Experience, ITU.
27. Usability of a Multimodal Video Game to Improve Navigation Skills for Blind Children10.1145/1857920.1857924 ↗
28. Multimodal integration10.1109/93.556457 ↗
29. A Human-Centered Taxonomy of Interaction Modalities and Devices10.1093/iwc/iwz003 ↗
30. Nizam, S.S.M., Abidin, R.Z., Hashim, N.C., Chun, M., Arshad, H., and Majid, N.A.A. (2018). A Review of Multimodal Interaction Technique in Augmented Reality Environment. Int. J. Adv. Sci. Eng. Inf. Technol., 8.10.18517/ijaseit.8.4-2.6824 ↗
31. A review on multimodal interaction in Mixed Reality Environment10.1088/1757-899X/551/1/012049 ↗
32. Kitchenham, B., and Charters, S. (2007). Guidelines for Performing Systematic Literature Reviews in Software Engineering, Keele University. Technical Report.
33. Seitz, A., Henze, D., Nickles, J., Sauer, M., and Bruegge, B. (2018, January 23–26). Augmenting the industrial Internet of Things with Emojis. Proceedings of the 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC), Barcelona, Spain.10.1109/FMEC.2018.8364073 ↗
34. Sahinel, D., Akpolat, C., Gorur, O.C., and Sivrikaya, F. (2019, January 15–18). Integration of Human Actors in IoT and CPS Landscape. Proceedings of the 2019 IEEE 5th World Forum on Internet of Things (WF-IoT), Limerick, Ireland.10.1109/WF-IoT.2019.8767276 ↗
35. Leppanen, T., Heikkinen, A., Karhu, A., Harjula, E., Riekki, J., and Koskela, T. (2014, January 10–12). Augmented Reality Web Applications with Mobile Agents in the Internet of Things. Proceedings of the 2014 Eighth International Conference on Next Generation Mobile Apps, Services and Technologies, Oxford, UK.10.1109/NGMAST.2014.24 ↗
36. Using Augmented Reality and Internet of Things to improve accessibility of people with motor disabilities in the context of Smart Cities10.1016/j.future.2016.11.030 ↗
37. Agrawal, D., Mane, S.B., Pacharne, A., and Tiwari, S. (2018, January 11–12). IoT Based Augmented Reality System of Human Heart: An Android Application. Proceedings of the 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India.10.1109/ICOEI.2018.8553807 ↗
38. Augmented and virtual reality based monitoring and safety system: A prototype IoT platform10.1016/j.jnca.2017.03.022 ↗
39. Augmented Reality Enabled IoT Services for Environmental Monitoring Utilising Serious Gaming Concept
40. An Augmented Reality Prototype for supporting IoT-based Educational Activities for Energy-efficient School Buildings10.1016/j.entcs.2019.04.012 ↗
41. Simões, B., De Amicis, R., Barandiaran, I., and Posada, J. (2018). X-Reality System Architecture for Industry 4.0 Processes. Multimodal Technol. Interact., 2.10.3390/mti2040072 ↗
42. Cho, K., Jang, H., Park, L.W., Kim, S., and Park, S. (2019, January 11–13). Energy Management System Based on Augmented Reality for Human-Computer Interaction in a Smart City. Proceedings of the 2019 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.10.1109/ICCE.2019.8662045 ↗
43. He, Y., Sawada, I., Fukuda, O., Shima, R., Yamaguchi, N., and Okumura, H. (2018, January 15–19). Development of an evaluation system for upper limb function using AR technology. Proceedings of the Genetic and Evolutionary Computation Conference Companion on—GECCO’18, Kyoto, Japan.10.1145/3205651.3208256 ↗
44. Huo, K., Cao, Y., Yoon, S.H., Xu, Z., Chen, G., and Ramani, K. (2018, January 21–26). Scenariot: Spatially Mapping Smart Things Within Augmented Reality Scenes. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems—CHI’18, Montreal QC, Canada.10.1145/3173574.3173793 ↗
45. An Augmented Reality/Internet of Things Prototype for Just-in-time Astronaut Training10.1007/978-3-319-57987-0_20 ↗
46. AR Enabled System for Cultural Heritage Monitoring and Preservation10.1007/978-3-030-01168-0_50 ↗
47. Phupattanasilp, P., and Tong, S.R. (2019). Augmented Reality in the Integrative Internet of Things (AR-IoT): Application for Precision Farming. Sustainability, 11.10.3390/su11092658 ↗
48. Cao, Y., Xu, Z., Li, F., Zhong, W., Huo, K., and Ramani, K.V. (2019, January 23–28). Ra: An In-Situ Visual Authoring System for Robot-IoT Task Planning with Augmented Reality. Proceedings of the 2019 on Designing Interactive Systems Conference—DIS’19, San Diego, CA, USA.10.1145/3322276.3322278 ↗
49. de Belen, R.A.J., Bednarz, T., and Favero, D.D. (2019, January 14–16). Integrating Mixed Reality and Internet of Things as an Assistive Technology for Elderly People Living in a Smart Home. Proceedings of the 17th International Conference on Virtual-Reality Continuum and its Applications in Industry, Brisbane, QLD, Australia.10.1145/3359997.3365742 ↗
50. ARIoT: Scalable augmented reality framework for interacting with Internet of Things appliances everywhere10.1109/TCE.2016.7613201 ↗
51. Purmaissur, J.A., Towakel, P., Guness, S.P., Seeam, A., and Bellekens, X.A. (2018, January 6–7). Augmented-Reality Computer-Vision Assisted Disaggregated Energy Monitoring and IoT Control Platform. Proceedings of the 2018 International Conference on Intelligent and Innovative Computing Applications (ICONIC), Plaine Magnien, Mauritius.10.1109/ICONIC.2018.8601199 ↗
52. Stefanidi, E., Foukarakis, M., Arampatzis, D., Korozi, M., Leonidis, A., and Antona, M. (2019). ParlAmI: A Multimodal Approach for Programming Intelligent Environments. Technologies, 7.10.3390/technologies7010011 ↗
53. Ten myths of multimodal interaction10.1145/319382.319398 ↗
54. Fuhl, W., Santini, T., Kasneci, G., and Kasneci, E. (2016). PupilNet: Convolutional Neural Networks for Robust Pupil Detection. arXiv.
55. Brain–Computer Interfaces for Multimodal Interaction: A Survey and Principles10.1080/10447318.2011.582022 ↗
56. Gorzkowski, S., and Sarwas, G. (2019, January 26–29). Exploitation of EMG Signals for Video Game Control. Proceedings of the 2019 20th International Carpathian Control Conference (ICCC), Krakow-Wieliczka, Poland.10.1109/CarpathianCC.2019.8765983 ↗
57. Liao, S.C., Wu, F.G., and Feng, S.H. (2019, January 2–5). Playing games with your mouth: Improving gaming experience with EMG supportive input device. Proceedings of the International Association of Societies of Design Research Conference, Manchester, UK.
58. Development and Testing of a Small-Size Olfactometer for the Perception of Food and Beverages in Humans10.3389/fdigh.2018.00007 ↗
59. Digital Lollipop: Studying Electrical Stimulation on the Human Tongue to Simulate Taste Sensations10.1145/2996462 ↗
60. Shifty: A Weight-Shifting Dynamic Passive Haptic Proxy to Enhance Object Perception in Virtual Reality10.1109/TVCG.2017.2656978 ↗
61. Hussain, I., Meli, L., Pacchierotti, C., Salvietti, G., and Prattichizzo, D. (2015, January 22–26). Vibrotactile haptic feedback for intuitive control of robotic extra fingers. Proceedings of the 2015 IEEE World Haptics Conference (WHC), Evanston, IL, USA.10.1109/WHC.2015.7177744 ↗
62. Al-Jabi, M., and Sammaneh, H. (2018, January 28–30). Toward Mobile AR-based Interactive Smart Parking System. Proceedings of the 2018 IEEE 20th International Conference on High Performance Computing and Communications, IEEE 16th International Conference on Smart City, IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Exeter, UK.10.1109/HPCC/SmartCity/DSS.2018.00209 ↗
63. Kim, J.C., Lindberg, R.S.N., Laine, T.H., Faarinen, E.C., Troyer, O.D., and Nygren, E. (2019, January 21–22). Multidisciplinary Development Process of a Story-based Mobile Augmented Reality Game for Learning Math. Proceedings of the 2019 17th International Conference on Emerging eLearning Technologies and Applications (ICETA), Smokovec, Slovakia.10.1109/ICETA48886.2019.9039962 ↗