A Review on Mixed Reality: Current Trends, Challenges and Prospects
Source abstract
Abstract
Currently, new technologies have enabled the design of smart applications that are used as decision-making tools in the problems of daily life. The key issue in designing such an application is the increasing level of user interaction. Mixed reality (MR) is an emerging technology that deals with maximum user interaction in the real world compared to other similar technologies. Developing an MR application is complicated, and depends on the different components that have been addressed in previous literature. In addition to the extraction of such components, a comprehensive study that presents a generic framework comprising all components required to develop MR applications needs to be performed. This review studies intensive research to obtain a comprehensive framework for MR applications. The suggested framework comprises five layers: the first layer considers system components; the second and third layers focus on architectural issues for component integration; the fourth layer is the application layer that executes the architecture; and the fifth layer is the user interface layer that enables user interaction. The merits of this study are as follows: this review can act as a proper resource for MR basic concepts, and it introduces MR development steps and analytical models, a simulation toolkit, system types, and architecture types, in addition to practical issues for stakeholders such as considering MR different domains.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Augmented Reality Applications · Interactive and Immersive Displays · Virtual Reality Applications and Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Computer Science and Engineering, Sejong University, Seoul 143-747, Korea
Geoinformation Technology, Center of Excellence, Faculty of Geomatics Engineering, K. N. Toosi University of Technology, Tehran 19697, Iran
Funding
MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program
References (100+)
Show source-provided references
1. Mobile Fire Evacuation System for Large Public Buildings Based on Artificial Intelligence and IoT10.1109/ACCESS.2019.2915241 ↗
2. Hoenig, W., Milanes, C., Scaria, L., Phan, T., Bolas, M., and Ayanian, N. (October, January 28). Mixed reality for robotics. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany.
3. A method of creating a transparent space based on mixed reality10.1007/s10015-006-0400-1 ↗
4. The impact of virtual, augmented and mixed reality technologies on the customer experience10.1016/j.jbusres.2018.10.050 ↗
5. Costanza, E., Kunz, A., and Fjeld, M. (2009). Mixed reality: A survey. Human Machine Interaction, Springer.10.1007/978-3-642-00437-7_3 ↗
6. De Guzman, J.A., Thilakarathna, K., and Seneviratne, A. (2018). Security and privacy approaches in mixed reality: A literature survey. arXiv.10.1145/3359626 ↗
7. State-of-the-Art Review on Mixed Reality Applications in the AECO Industry10.1061/(ASCE)CO.1943-7862.0001749 ↗
8. Bach, C., and Scapin, D.L. (2004). Obstacles and perspectives for evaluating mixed reality systems usability. Acte du Workshop MIXER, IUI-CADUI, Citeseer.
9. Jacobs, K., and Loscos, C. (2006). Classification of illumination methods for mixed reality. Computer Graphics Forum, Wiley Online Library.10.1111/j.1467-8659.2006.00816.x ↗
10. Exploring mobile mixed reality in healthcare higher education: A systematic review10.25304/rlt.v26.2131 ↗
11. Murphy, K.M., Cash, J., and Kellinger, J.J. (2018). Learning with avatars: Exploring mixed reality simulations for next-generation teaching and learning. Handbook of Research on Pedagogical Models for Next-Generation Teaching and Learning, IGI Global.10.4018/978-1-5225-3873-8.ch001 ↗
12. Reality check: Basics of augmented, virtual, and mixed reality10.1080/02763869.2017.1293987 ↗
13. Ohta, Y., and Tamura, H. (2014). Mixed Reality: Merging Real and Virtual Worlds, Springer Publishing.
14. MemHolo: Mixed reality experiences for subjects with Alzheimer’s disease10.1007/s11042-018-7089-8 ↗
15. Exploring the design space of immersive urban analytics10.1016/j.visinf.2017.11.002 ↗
16. Florins, M., Trevisan, D.G., and Vanderdonckt, J. (2005). The continuity property in mixed reality and multiplatform systems: A comparative study. Computer-Aided Design of User Interfaces IV, Springer.10.1007/1-4020-3304-4_26 ↗
17. From E-Learning to M-Learning-The use of mixed reality games as a new educational paradigm10.3991/ijim.v5i2.1463 ↗
18. A taxonomy of mixed reality visual displays
19. Setting the best view of a virtual teacher in a mixed reality physical-task learning support system10.1016/j.jss.2012.08.060 ↗
20. Virtuality Continuum’s State of the Art10.1016/j.procs.2013.11.032 ↗
21. Supporting trembling hand typing using optical see-through mixed reality10.1109/ACCESS.2017.2711058 ↗
22. 3iVClass: A new classification method for virtual, augmented and mixed realities10.1016/j.procs.2018.10.180 ↗
23. Rosenberg, L.B. (1992). The Use of Virtual Fixtures as Perceptual Overlays to Enhance Operator Performance in Remote Environments. Stanford University Ca Center for Design Research, Stanford University.10.1037/e477812004-001 ↗
24. Virtual reality, augmented reality, mixed reality, and the marine conservation movement10.1002/aqc.2820 ↗
25. Potemin, I.S., Zhdanov, A., Bogdanov, N., Zhdanov, D., Livshits, I., and Wang, Y. (2018, January 5). Analysis of the visual perception conflicts in designing mixed reality systems. Proceedings of the Optical Design and Testing VIII, International Society for Optics and Photonics, Beijing, China.
26. Claydon, M. (2015). Alternative Realities: From Augmented Reality to Mobile Mixed Reality. [Master’s Thesis, University of Tampere].
27. Kress, B.C., and Cummings, W.J. (2017). 11–1: Invited paper: Towards the ultimate mixed reality experience: HoloLens display architecture choices. SID Symposium Digest of Technical Papers, Wiley Online Library.10.1002/sdtp.11586 ↗
28. Task-driven latent active correction for physics-inspired input method in near-field mixed reality applications10.1002/jsid.728 ↗
29. Calibration of fish-eye camera for acquisition of spherical image10.1002/scj.20631 ↗
30. Calibration of RGBD camera and cone-beam CT for 3D intra-operative mixed reality visualization10.1007/s11548-016-1396-1 ↗
31. Groves, L., Li, N., Peters, T.M., and Chen, E.C. (2019). Towards a Mixed-Reality First Person Point of View Needle Navigation System. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-030-32254-0_28 ↗
32. Steed, A., MacColl, J., Randell, C., Brown, B., Chalmers, M., and Greenhalgh, C. (2004, January 14–16). Models of space in a mixed-reality system. Proceedings of the Eighth International Conference on Information Visualisation, IV 2004, London, UK.
33. Xing, H., Bao, X., Zhang, F., Gai, W., Qi, M., Liu, J., Shi, Y., De Melo, G., Yang, C., and Meng, X. (2019, January 23–27). Rotbav: A Toolkit for Constructing Mixed Reality Apps with Real-Time Roaming in Large Indoor Physical Spaces. Proceedings of the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Osaka, Japan.10.1109/VR.2019.8798050 ↗
34. Nagata, K., Masuko, S., and Hoshino, J. (2018). Construction of Mixed Reality Story Environment Based on Real Space Shape. International Conference on Entertainment Computing, Springer.10.1007/978-3-319-99426-0_27 ↗
35. A high-resolution model for soft tissue deformation based on point primitives10.1016/j.cmpb.2017.06.013 ↗
36. Kakuta, T., Vinh, L.B., Kawakami, R., Oishi, T., and Ikeuchi, K. (, January October). Detection of moving objects and cast shadows using a spherical vision camera for outdoor mixed reality. Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology, New York, NY, USA.10.1145/1450579.1450626 ↗
37. Chen, L., Tang, W., John, N., Wan, T.R., and Zhang, J.J. (2018). Context-Aware Mixed Reality: A Framework for Ubiquitous Interaction. arXiv.
38. Hoppenstedt, B., Kammerer, K., Reichert, M., Spiliopoulou, M., and Pryss, R. (2019). Convolutional Neural Networks for Image Recognition in Mixed Reality Using Voice Command Labeling. Augmented Reality, Virtual Reality, and Computer Graphics, Springer.10.1007/978-3-030-25999-0_6 ↗
39. Ambient Browser: Mobile Mixed Reality System Based on Ambient Intelligence10.7763/IJFCC.2012.V1.41 ↗
40. Bonfert, M., Lehne, I., Morawe, R., Cahnbley, M., Zachmann, G., and Schöning, J. (2017, January 8–10). Augmented invaders: A mixed reality multiplayer outdoor game. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, Gothenburg, Sweden.10.1145/3139131.3141208 ↗
41. Mathews, Z.I., Badia, S.B., and Paul, F. (2007, January 25–27). Are You There? Active Attention for Person Tracking in Mixed Reality Environments. Proceedings of the 10th Annual International Workshop on Presence (PRESENCE 2007), Barcelona, Spain.
42. Construction of a computer-simulated mixed reality environment for virtual factory layout planning10.1016/j.compind.2010.07.001 ↗
43. Ammari, K.E., and Hammad, A. (2014, January 23–25). Collaborative BIM-based markerless mixed reality framework for facilities maintenance. Proceedings of the Computing in Civil and Building Engineering, Orlando, FL, USA.10.1061/9780784413616.082 ↗
44. A virtual try-on system in augmented reality using RGB-D cameras for footwear personalization10.1016/j.jmsy.2014.05.006 ↗
45. Mixed Reality for on-site self-instruction and self-inspection with Building Information Models10.1016/j.procir.2018.03.160 ↗
46. Bim mixed reality tool for the inspection of heritage buildings10.5194/isprs-annals-IV-2-W6-25-2019 ↗
47. A Marker-Less Registration Approach for Mixed Reality–Aided Maxillofacial Surgery: A Pilot Evaluation10.1007/s10278-019-00272-6 ↗
48. Azimi, E., Qian, L., Navab, N., and Kazanzides, P. (2017). Alignment of the Virtual Scene to the Tracking Space of a Mixed Reality Head-Mounted Display. arXiv.
49. Evaluation of a haptic mixed reality system for interactions with a virtual control panel10.1162/105474605775196562 ↗
50. Interactive chroma keying for mixed reality10.1002/cav.305 ↗
51. Deep spherical harmonics light probe estimator for mixed reality games10.1016/j.cag.2018.09.003 ↗
52. Reflective and refractive objects for mixed reality10.1109/TVCG.2013.39 ↗
53. Simulated real-time soft shadow in mixed reality using fuzzy logic10.1007/3DRes.03(2013)1 ↗
54. Rendering techniques for mixed reality10.1007/s11554-009-0137-x ↗
55. Proposal of an Information System for an Adaptive Mixed Reality System for Archaeological Sites10.1016/j.protcy.2014.10.117 ↗
56. Eddine, A.N., and Junjun, P. (2018). Geospatial Data Holographic Rendering Using Windows Mixed Reality. International Conference on E-Learning and Games, Springer.
57. Roxas, M., Hori, T., Fukiage, T., Okamoto, Y., and Oishi, T. (December, January 28). Occlusion handling using semantic segmentation and visibility-based rendering for mixed reality. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology, Tokyo, Japan.
58. Sandor, C., Dey, A., Cunningham, A., Barbier, S., Eck, U., Urquhart, D., Marner, M.R., Jarvis, G., and Rhee, S. (2010, January 19–22). Egocentric space-distorting visualizations for rapid environment exploration in mobile mixed reality. Proceedings of the 2010 IEEE Virtual Reality Conference (VR), Orlando, FL, USA.10.1109/VR.2010.5444815 ↗
59. DVV: A taxonomy for mixed reality visualization in image guided surgery
60. Ai, Z., Livingston, M.A., and Moskowitz, I.S. (2016, January 7–10). Real-time unmanned aerial vehicle 3D environment exploration in a mixed reality environment. Proceedings of the 2016 International Conference on Unmanned Aircraft Systems (ICUAS), Arlington, VA, USA.10.1109/ICUAS.2016.7502588 ↗
61. Ritsos, P.D., Mearman, J.W., Jackson, J.R., and Roberts, J.C. (2017, January 1–6). Synthetic Visualizations in Web-based Mixed Reality. Proceedings of the Immersive Analytics: Exploring Future Visualization and Interaction Technologies for Data Analytics Workshop, IEEE Conference on Visualization (VIS), Phoenix, AZ, USA.
62. A mixed-reality system for broadcasting sports video to mobile devices10.1109/MMUL.2010.24 ↗
63. Lee, R., Kwon, Y.-J., and Sumiya, K. (2009, January 15–18). Layer-based media integration for mobile mixed-reality applications. Proceedings of the Third International Conference on Next Generation Mobile Applications, Services and Technologies, Cardiff, Wales.10.1109/NGMAST.2009.90 ↗
64. Kim, H., Yoon, H., Choi, A., Baek, W., Lee, I., Kim, D., and Woo, W. (2011, January 17–19). Data markup representation for mixed reality contents. Proceedings of the International AR Standards Meeting-February, Barcelona, Spain.
65. Elrawi, O.M. (2017, January 17–19). The Use of Mixed-Realities Techniques for the Representation of Islamic Cultural Heritage. Proceedings of the 2017 International Conference on Machine Vision and Information Technology (CMVIT), Singapore.10.1109/CMVIT.2017.16 ↗
66. Panahi, A. (2017). Big Data Visualization Platform for Mixed Reality. [Master’s Thesis, Virginia Commonwealth University].
67. Mixed reality applications in urban environments10.1023/B:BTTJ.0000047123.94280.3a ↗
68. Abawi, D.F., Dörner, R., Haller, M., and Zauner, J. (2004, January 15–16). Efficient mixed reality application development. Proceedings of the 1st European Conference on Visual Media Production (CVMP), London, UK.
69. Mixed reality in learning factories10.1016/j.promfg.2018.04.009 ↗
70. Liarokapis, F., and Newman, R.M. (2007, January 22–24). Design experiences of multimodal mixed reality interfaces. Proceedings of the 25th Annual ACM International Conference on Design of Communication, El Paso, TX, USA.10.1145/1297144.1297152 ↗
71. Visuo-haptic mixed reality with unobstructed tool-hand integration10.1109/TVCG.2012.107 ↗
72. Tangible mixed reality for remote design review: A study understanding user perception and acceptance10.1186/2213-7459-1-8 ↗
73. Piumsomboon, T., Lee, G.A., and Billinghurst, M. (2018). Snow Dome: A Multi-Scale Interaction in Mixed Reality Remote Collaboration. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, ACM.10.1145/3170427.3186495 ↗
74. William, L., Boo, I., and de Souza, R. (2018, January 4–7). Embedding Mixed Reality in Humanitarian Logistics Gaming. Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), Wollongong, Australia.10.1109/TALE.2018.8615265 ↗
75. Contributions of mixed reality in a calligraphy learning task: Effects of supplementary visual feedback and expertise on cognitive load, user experience and gestural performance10.1016/j.chb.2017.05.006 ↗
76. Mixed-reality learning environments: Integrating mobile interfaces with laboratory test-beds10.1016/j.compedu.2017.02.009 ↗
77. A mobile mixed-reality environment for children’s storytelling using a handheld projector and a robot10.1109/TLT.2011.13 ↗
78. Olorunfemi, A. (2018). Assessing the Feasibility of Applying Mixed Reality in Enhancing Construction Site Safety Communication. [Ph.D. Thesis, West Virginia University].
79. Mixed Reality Guided Radiofrequency Needle Placement: A Pilot Study10.1109/ACCESS.2018.2843378 ↗
80. Lemordant, J., Michel, T., and Razafinahazo, M. (2013, January 21–23). Mixed Reality Browsers and Pedestrian Navigation in Augmented Cities. Proceedings of the Graphical Web Conference, San Francisco, CA, USA.
81. Reilly, D., Salimian, M., MacKay, B., Mathiasen, N., Edwards, W.K., and Franz, J. (2014, January 17–20). SecSpace: Prototyping usable privacy and security for mixed reality collaborative environments. Proceedings of the 2014 ACM SIGCHI Symposium on Engineering Interactive Computing Systems, Rome, Italy.10.1145/2607023.2607039 ↗
82. Gardner, M., Scott, J., and Horan, B. (2008, January 20–21). Reflections on the use of Project Wonderland as a mixed-reality environment for teaching and learning. Proceedings of the Researching Learning in Virtual Environments International Conference, The Open University, Milton Keynes, UK.
83. Du, R., Li, D., and Varshney, A. (2019, January 4–9). Geollery: A Mixed Reality Social Media Platform. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, Glasgow, UK.10.1145/3290605.3300915 ↗
84. De Guzman, J.A., Thilakarathna, K., and Seneviratne, A. (2019). A First Look into Privacy Leakage in 3D Mixed Reality Data. European Symposium on Research in Computer Security, Springer.10.1007/978-3-030-29959-0_8 ↗
85. A mixed reality environment for collaborative product design and development10.1016/j.cirp.2009.03.020 ↗
86. Ogino, T., Matsuda, Y., Kawabata, D., Yamazaki, K., Kimura, A., and Shibata, F. (2014, January 7–10). A distributed framework for creating mobile mixed reality systems. Proceedings of the 2014 IEEE 3rd Global Conference on Consumer Electronics (GCCE), Tokyo, Japan.10.1109/GCCE.2014.7031145 ↗
87. Selonen, P., Belimpasakis, P., and You, Y. (2010). Experiences in building a RESTful mixed reality web service platform. International Conference on Web Engineering, Springer.10.1145/1798354.1798387 ↗
88. Wendrich, R.E. (2014, January 17–20). Mixed Reality Tools for Playful Representation of Ideation, Conceptual Blending and Pastiche in Design and Engineering. Proceedings of the ASME 2014 International Design Engineering Technical Conference and Computers and Information in Engineering, Buffalo, NY, USA.10.1115/DETC2014-34926 ↗
89. Stantchev, V. (2009). Enhancing health care services with mixed reality systems. The Engineering of Mixed Reality Systems Berlin, Springer.10.1007/978-1-84882-733-2_17 ↗
90. Hamza-Lup, F.G. (2018). Sensors in Distributed Mixed Reality Environments. arXiv.
91. Lee, S., Lee, G., Choi, G., Roh, B.-H., and Kang, J. (2019, January 11–13). Integration of OneM2M-based IoT Service Platform and Mixed Reality Device. Proceedings of the 2019 IEEE International Conference on Consumer Electronics (ICCE), Las Vegas, NV, USA.10.1109/ICCE.2019.8662008 ↗
92. Mixar mobile prototype: Visualizing virtually reconstructed ancient structures in situ10.1016/j.procs.2015.08.638 ↗
93. A combined mixed reality and networked home approach to improving user interaction with consumer electronics10.1109/TCE.2011.5735494 ↗
94. Multimodal mixed reality interfaces for visualizing digital heritage10.1260/1478-0771.5.2.322 ↗
95. Corral-Soto, E.R., Tal, R., Wang, L., Persad, R., Chao, L., Solomon, C., Hou, B., Sohn, G., and Elder, J.H. (2012, January 28–30). 3D town: The automatic urban awareness project. Proceedings of the 2012 Ninth Conference on Computer and Robot Vision, Toronto, ON, Canada.10.1109/CRV.2012.64 ↗
96. Newman, J., Bornik, A., Pustka, D., Echtler, F., Huber, M., Schmalstieg, D., and Klinker, G. (2007, January 10–14). Tracking for distributed mixed reality environments. Proceedings of the Workshop on Trands and Issues in Tracking for Virtual Environments at the IEEE Virtual Reality Conference (VR’07), Charlotte, NC, USA.
97. User perspectives on mixed reality tabletop visualization for face-to-face collaborative design review10.1016/j.autcon.2007.07.002 ↗
98. Mixed reality in education, entertainment, and training10.1109/MCG.2005.139 ↗
99. Pomi, A., Mannitt, G., Wald, I., and Slusallek, P. (2003, January 19–21). Sttreaming Video Textures for Mixed Reality Applications in Interactive Ray Tracing Environments. Proceedings of the Vision, Modeling, and Visualization Conference 2003 (VMV 2003), München, Germany.
100. Pollock, B., Winer, E., Gilbert, S., and de la Cruz, J. (2012, January 22–26). LVC Interaction within a Mixed-Reality Training System. Proceedings of the Engineering Reality of Virtual Reality 2012, International Society for Optics and Photonics, Santa Clara, CA, USA.10.1117/12.912193 ↗