Virtual Object Manipulation by Combining Touch and Head Interactions for Mobile Augmented Reality
Source abstract
Abstract
This paper proposes an interaction method to conveniently manipulate a virtual object by combining touch interaction and head movements for a head-mounted display (HMD), which provides mobile augmented reality (AR). A user can conveniently manipulate a virtual object with touch interaction recognized from the inertial measurement unit (IMU) attached to the index finger’s nail and head movements tracked by the IMU embedded in the HMD. We design two interactions that combine touch and head movements, to manipulate a virtual object on a mobile HMD. Each designed interaction method manipulates virtual objects by controlling ray casting and adjusting widgets. To evaluate the usability of the designed interaction methods, a user evaluation is performed in comparison with the hand interaction using Hololens. As a result, the designed interaction method receives positive feedback that virtual objects can be manipulated easily in a mobile AR environment.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Interactive and Immersive Displays · Augmented Reality Applications · Tactile and Sensory Interactions
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
NT-IT (HCI & Robotics), Korea University of Science and Technology, Gajeong-ro, Yuseong-gu, Daejeon 34113, Korea
Center for Intelligent & Interactive Robotics, Korea Institute of Science and Technology, 5, Hwarang-ro 14-gil, Seongbuk-gu, Seoul 02792, Korea
Funding
National Research Foundation of Korea
References (43)
Show source-provided references
1. LaViola, J.J., Kruijff, E., McMahan, R.P., Bowman, D., and Poupyrev, I.P. (2017). 3D User Interfaces: Theory and Practice, Addison-Wesley Professional.
2. (2019, June 23). Google Glass. Available online: https://www.google.com/glass/start/.
3. Leap, M. (2019, June 23). Magic Leap. Available online: https://www.magicleap.com/.
4. (2019, June 23). Microsoft Hololens. Available online: https://www.microsoft.com/en-us/p/microsoft-hololens-development-edition/8xf18pqz17ts?activetab=pivot:overviewtab.
5. (2019, June 23). Sony Smarteyeglass SED E1. Available online: https://developer.sony.com/develop/smarteyeglass-sed-e1/.
6. Joolee, J., Uddin, M., Khan, J., Kim, T., and Lee, Y.K. (2018). A Novel Lightweight Approach for Video Retrieval on Mobile Augmented Reality Environment. Appl. Sci., 8.10.3390/app8101860 ↗
7. Interior design in augmented reality environment
8. Nóbrega, R., and Correia, N. (2011). CHI’11 Extended Abstracts on Human Factors in Computing Systems. Design Your Room: Adding Virtual Objects to a Real Indoor Scenario, ACM.10.1145/1979742.1979851 ↗
9. Gugenheimer, J., Dobbelstein, D., Winkler, C., Haas, G., and Rukzio, E. (2016, January 16–19). Face Touch: Enabling Touch Interaction in Display fixed Uis for Mobile Virtual Reality. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, Japan.10.1145/2984511.2984576 ↗
10. Tung, Y.C., Hsu, C.Y., Wang, H.Y., Chyou, S., Lin, J.W., Wu, P.J., Valstar, A., and Chen, M.Y. (2015, January 18–23). User-Defined Game Input for Smart Glasses in Public Space. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.10.1145/2702123.2702214 ↗
11. Mendes, D., Relvas, F., Ferreira, A., and Jorge, J. (2016, January 2–4). The Benefits of DOF Separation in Mid-Air 3d Object Manipulation. Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, Munich, Germany.10.1145/2993369.2993396 ↗
12. Bader, T., Vogelgesang, M., and Klaus, E. (2009, January 2–4). Multimodal Integration of Natural Gaze Behavior for Intention Recognition During Object Manipulation. Proceedings of the 2009 International Conference on Multimodal Interfaces, Cambridge, UK.10.1145/1647314.1647350 ↗
13. Fukazawa, R., Takashima, K., Shoemaker, G., Kitamura, Y., Itoh, Y., and Kishino, F. (2010, January 20–21). Comparison of multimodal interactions in perspective-corrected multi-display environment. Proceedings of the 2010 IEEE Symposium on 3D User Interfaces (3DUI), Waltham, MA, USA.10.1109/3DUI.2010.5444711 ↗
14. Oh, J.Y., Lee, J., Lee, J.H., and Park, J.H. (2017, January 9–14). Anywheretouch: Finger tracking method on arbitrary surface using nailed-mounted imu for mobile hmd. Proceedings of the International Conference on Human-Computer Interaction, Vancouver, BC, Canada.10.1007/978-3-319-58750-9_26 ↗
15. 3D Selection Techniques for Mobile Augmented Reality Head-Mounted Displays
16. Mine, M.R. (1995). Virtual Environment Interaction Techniques, UNC Chapel Hill CS Dept.
17. (2019, June 23). Microsoft Bounding Box Default Handle Sytle. Available online: https://github.com/microsoft/MixedRealityToolkit-Unity/blob/mrtk_release/Documentation/README_HandInteractionExamples.md.
18. Testbed evaluation of virtual environment interaction techniques10.1162/105474601750182333 ↗
19. Hoppe, A.H., van de Camp, F., and Stiefelhagen, R. (2017, January 9–14). Interaction with three dimensional objects on diverse input and output devices: A survey. Proceedings of the International Conference on Human-Computer Interaction, Vancouver, BC, Canada.10.1007/978-3-319-58750-9_18 ↗
20. Chen, M., Mountford, S.J., and Sellen, A. (1988). A Study in Interactive 3-D Rotation Using 2-D Control Devices, ACM.10.1145/54852.378497 ↗
21. Conner, B.D., Snibbe, S.S., Herndon, K.P., Robbins, D.C., Zeleznik, R.C., and Van Dam, A. (1990, January 10–11). Three-dimensional widgets. Proceedings of the 1992 Symposium on Interactive 3D graphics, San Diego, CA, USA.10.1145/147156.147199 ↗
22. Houde, S. (May, January USA). Iterative design of an interface for easy 3-D direct manipulation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Monterey, CA.
23. Shoemake, K. (1992). ARCBALL: A User interface for specifying three-dimensional orientation using a mouse. Graphics Interface, Canadian Information Processing Society.
24. Techologies, U. (2019, June 23). Unity3d. Available online: https://unity.com/.
25. (2019, June 23). Autodesk Maya. Available online: https://www.autodesk.com/products/maya/overview.
26. (2019, June 23). Blender Blender. Available online: https://www.blender.org/.
27. Mendes, D., Caputo, F.M., Giachetti, A., Ferreira, A., and Jorge, J. (2019). Computer Graphics Forum. A Survey on 3d Virtual Object Manipulation: From the Desktop to Immersive Virtual Environments, Wiley Online Library.10.1111/cgf.13390 ↗
28. Cohé, A., Dècle, F., and Hachet, M. (2011, January 7–12). tBox: A 3d Transformation Widget Designed for Touch-Screens. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Vancouver, BC, Canada.10.1145/1978942.1979387 ↗
29. Mendes, D., Lopes, P., and Ferreira, A. (2011, January 8–11). Hands-on Interactive Tabletop Lego Application. Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology, Lisbon, Portugal.10.1145/2071423.2071447 ↗
30. Au, O.K.C., Tai, C.L., and Fu, H. (2012). Computer Graphics Forum. Multitouch Gestures for Constrained Transformation of 3d Objects, Wiley Online Library.10.1111/j.1467-8659.2012.03044.x ↗
31. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality10.1109/TVCG.2018.2794222 ↗
32. Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., and Wei, Y. (2015, January 18–23). Accurate, Robust, and Flexible Real-Time Hand Tracking. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea.10.1145/2702123.2702179 ↗
33. Frees, S., and Kessler, G.D. (2005, January 12–16). Precise and rapid interaction through scaled manipulation in immersive virtual environments. Proceedings of the IEEE Proceedings Virtual Reality (VR 2005), Bonn, Germany.
34. Poupyrev, I., Billinghurst, M., Weghorst, S., and Ichikawa, T. (1996, January 6–8). The go-go interaction technique: Non-linear mapping for direct manipulation in VR. Proceedings of the ACM Symposium on User Interface Software and Technology, Seattle, WA, USA.10.1145/237091.237102 ↗
35. An Evaluation of Techniques for Grabbing and Manipulating Remote Objects in Immersive Virtual Environments10.1145/253284.253301 ↗
36. Nancel, M., Chapuis, O., Pietriga, E., Yang, X.D., Irani, P.P., and Beaudouin-Lafon, M. (May, January 27). High-precision pointing on large wall displays using small handheld devices. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.
37. A usability study of multimodal input in an augmented reality environment10.1007/s10055-013-0230-0 ↗
38. Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., and Billinghurst, M. (2014, January 10–12). Grasp-Shell vs gesture-speech: A comparison of direct and indirect natural interaction techniques in augmented reality. Proceedings of the 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany.10.1109/ISMAR.2014.6948411 ↗
39. Ens, B., Byagowi, A., Han, T., Hincapié-Ramos, J.D., and Irani, P. (2016, January 15–16). Combining Ring Input with Hand Tracking for Precise, Natural Interaction with Spatial Analytic Interfaces. Proceedings of the 2016 Symposium on Spatial User Interaction, Tokyo, Japan.10.1145/2983310.2985757 ↗
40. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use10.1080/10447319509526110 ↗
41. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness10.1207/s15327108ijap0303_3 ↗
42. A survey of 3D object selection techniques for virtual environments10.1016/j.cag.2012.12.003 ↗
43. Review on cybersickness in applications and visual displays10.1007/s10055-016-0285-9 ↗