Silhouettes from Real Objects Enable Realistic Interactions with a Virtual Human in Mobile Augmented Reality
Source abstract
Abstract
Realistic interactions with real objects (e.g., animals, toys, robots) in an augmented reality (AR) environment enhances the user experience. The common AR apps on the market achieve realistic interactions by superimposing pre-modeled virtual proxies on the real objects in the AR environment. This way user perceives the interaction with virtual proxies as interaction with real objects. However, catering to environment change, shape deformation, and view update is not a trivial task. Our proposed method uses the dynamic silhouette of a real object to enable realistic interactions. Our approach is practical, lightweight, and requires no additional hardware besides the device camera. For a case study, we designed a mobile AR application to interact with real animal dolls. Our scenario included a virtual human performing four types of realistic interactions. Results demonstrated our method’s stability that does not require pre-modeled virtual proxies in case of shape deformation and view update. We also conducted a pilot study using our approach and reported significant improvements in user perception of spatial awareness and presence for realistic interactions with a virtual human.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Augmented Reality Applications · Robotics and Sensor-Based Localization · Virtual Reality Applications and Impacts
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Center for Artificial Intelligence, Korea Institute of Science and Technology (KIST), Seoul 02792, Korea
Department of Computer Science and Engineering, Korea University, Seoul 02841, Korea
Division of NT-IT, University of Science and Technology, Seoul 02792, Korea
LS2N, University of Nantes, 44200 Nantes, France
School of Computer Science and Engineering, Pusan National University, Busan 46241, Korea
Funding
National Research Council of Science and Technology
References (56)
Show source-provided references
1. A Survey of Augmented Reality10.1162/pres.1997.6.4.355 ↗
2. Preparing the HoloLens for user studies: An augmented reality interface for the spatial adjustment of holographic objects in 3D indoor environments10.1007/s42489-019-00025-z ↗
3. Ali, G., Le, H.Q., Kim, J., Hwang, S.W., and Hwang, J.I. (2019, January 1–3). Design of Seamless Multi-modal Interaction Framework for Intelligent Virtual Agents in Wearable Mixed Reality Environment. Proceedings of the 32nd International Conference on Computer Animation and Social Agents, Paris, France.10.1145/3328756.3328758 ↗
4. Sanches, S.R.R., Tokunaga, D.M., Silva, V.F., Sementille, A.C., and Tori, R. (2012, January 9–11). Mutual occlusion between real and virtual elements in Augmented Reality based on fiducial markers. Proceedings of the 2012 IEEE Workshop on the Applications of Computer Vision (WACV), Breckenridge, CO, USA.10.1109/WACV.2012.6163037 ↗
5. Kruijff, E., Swan, J.E., and Feiner, S. (2010, January 13–16). Perceptual issues in augmented reality revisited. Proceedings of the 2010 IEEE International Symposium on Mixed and Augmented Reality, Seoul, Korea.10.1109/ISMAR.2010.5643530 ↗
6. The Impacts of Visual Effects on User Perception With a Virtual Human in Augmented Reality Conflict Situations10.1109/ACCESS.2021.3062037 ↗
7. Kim, H., Kim, T., Lee, M., Kim, G.J., and Hwang, J.I. (2020, January 2–4). Don’t Bother Me: How to Handle Content-Irrelevant Objects in Handheld Augmented Reality. Proceedings of the 26th ACM Symposium on Virtual Reality Software and Technology, Ottawa, ON, Canada.10.1145/3385956.3418948 ↗
8. The effects of virtual human’s spatial and behavioral coherence with physical objects on social presence in AR10.1002/cav.1771 ↗
9. Milne, M., Luerssen, M.H., Lewis, T.W., Leibbrandt, R.E., and Powers, D.M.W. (2010, January 18–23). Development of a virtual agent based social tutor for children with autism spectrum disorders. Proceedings of the 2010 International Joint Conference on Neural Networks (IJCNN), Barcelona, Spain.10.1109/IJCNN.2010.5596584 ↗
10. Creating interactive virtual humans: Some assembly required10.1109/MIS.2002.1024753 ↗
11. Matsumura, K., and Sumi, Y. (2013, January 16–17). Poster: Puppetooner: A puppet-based system to interconnect real and virtual spaces for 3D animations. Proceedings of the 2013 IEEE Symposium on 3D User Interfaces (3DUI), Orlando, FL, USA.10.1109/3DUI.2013.6550227 ↗
12. Held, R., Gupta, A., Curless, B., and Agrawala, M. (2012, January 7–10). 3D Puppetry: A Kinect-Based Interface for 3D Animation. Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, Cambridge, MA, USA.10.1145/2380116.2380170 ↗
13. Tideland animal AR: Superimposing 3D animal models to user defined targets for augmented reality game10.14257/ijmue.2014.9.4.35 ↗
14. Igarashi, T., Matsuoka, S., and Tanaka, H. (1999, January 8–13). Teddy. Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques—SIGGRAPH ’99, Los Angeles, CA, USA.10.1145/311535.311602 ↗
15. Barakonyi, I., Psik, T., and Schmalstieg, D. (2004, January 5). Agents That Talk And Hit Back: Animated Agents in Augmented Reality. Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality, Arlington, VA, USA.
16. Taheri, A., Shahab, M., Meghdari, A., Alemi, M., Amoozandeh Nobaveh, A., Rokhi, Z., and Ghorbandaei Pour, A. (2018). Virtual Social Toys: A Novel Concept to Bring Inanimate Dolls to Life. International Conference on Social Robotics, Springer.10.1007/978-3-030-05204-1_28 ↗
17. Sesame Street, and Vuforia (2021, March 18). Sesame Workshop Prototype Playset Augmented Reality Vuforia. Available online: https://youtu.be/iR8uGxiX5ak.
18. GoonAR: A Bilingual Children Storybook through Augmented Reality Technology Using Unity with Vuforia Framework10.30534/ijatcse/2020/180932020 ↗
19. Tsvetkova, I., Kinaneva, D., Hristov, G., Raychev, J., and Zahariev, P. (2018, January 26–28). A complex workflow for development of interactive and impressive educational content using capabilities of animated augmented reality trends. Proceedings of the 2018 17th International Conference on Information Technology Based Higher Education and Training (ITHET), Olhao, Portugal.10.1109/ITHET.2018.8424776 ↗
20. PTC, and Vuforia (2021, March 18). Vuforia Engine: How to Create Model Targets. Available online: https://youtu.be/jbaUDMvv2Zw.
21. Kato, H., and Billinghurst, M. (1999). Marker tracking and HMD calibration for a video-based augmented reality conferencing system. Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR’99). IEEE Comput. Soc., 85–94.10.1109/IWAR.1999.803809 ↗
22. Geometrically-Correct Projection-Based Texture Mapping onto a Deformable Object10.1109/TVCG.2014.25 ↗
23. Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., and Theobalt, C. (2017, January 22–29). Real-Time Hand Tracking Under Occlusion From an Egocentric RGB-D Sensor. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.131 ↗
24. Tang, X., Hu, X., Fu, C.W., and Cohen-Or, D. (2020, January 20–23). GrabAR: Occlusion-aware Grabbing Virtual Objects in AR. Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, Minneapolis, MN, USA.10.1145/3379337.3415835 ↗
25. Azuma, R.T. (2017). Making Augmented Reality a Reality. Imaging and Applied Optics 2017 (3D, AIO, COSI, IS, MATH, pcAOP), Optical Society of America, OSA.10.1364/3D.2017.JTu1F.1 ↗
26. A framework for context immersion in mobile augmented reality10.1016/j.autcon.2012.10.020 ↗
27. Real-Time 3D Tracking and Reconstruction on Mobile Phones10.1109/TVCG.2014.2355207 ↗
28. Runz, M., Buffier, M., and Agapito, L. (2018, January 16–20). MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects. Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany.10.1109/ISMAR.2018.00024 ↗
29. Ozawa, T., Nakajima, Y., and Saito, H. (2019, January 28–29). Simultaneous 3D Tracking and Reconstruction of Multiple Moving Rigid Objects. Proceedings of the 2019 12th Asia Pacific Workshop on Mixed and Augmented Reality (APMAR), Ikoma, Japan.10.1109/APMAR.2019.8709158 ↗
30. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., and Fitzgibbon, A. (2011, January 26–29). KinectFusion: Real-time dense surface mapping and tracking. Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2011, Basel, Switzerland.10.1109/ISMAR.2011.6162880 ↗
31. Bastian, J., Ward, B., Hill, R., van den Hengel, A., and Dick, A. (2010, January 13–16). Interactive modelling for AR applications. Proceedings of the 2010 IEEE International Symposium on Mixed and Augmented Reality, Seoul, Korea.10.1109/ISMAR.2010.5643570 ↗
32. A semi-automatic method for resolving occlusion in augmented reality. Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662)
33. Mask R-CNN10.1109/TPAMI.2018.2844175 ↗
34. Chen, L.C., Papandreou, G., Schroff, F., and Adam, H. (2017). Rethinking Atrous Convolution for Semantic Image Segmentation. Comput. Vis. Pattern Recognit., Available online: http://arxiv.org/abs/1706.05587.
35. Zhao, H., Shi, J., Qi, X., Wang, X., and Jia, J. (2017, January 21–26). Pyramid Scene Parsing Network. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA.10.1109/CVPR.2017.660 ↗
36. The Pascal Visual Object Classes Challenge: A Retrospective10.1007/s11263-014-0733-5 ↗
37. Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016, January 27–30). The Cityscapes Dataset for Semantic Urban Scene Understanding. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.350 ↗
38. Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., and Hajishirzi, H. (2018, January 8–14). ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01249-6_34 ↗
39. Paszke, A., Chaurasia, A., Kim, S., and Culurciello, E. (2016). ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation. arXiv.
40. Oh, S.W., Lee, J.Y., Xu, N., and Kim, S.J. (2019, January 16–20). Fast User-Guided Video Object Segmentation by Interaction-and-Propagation Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA.10.1109/CVPR.2019.00539 ↗
41. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. (2014, January 23–28). Describing Textures in the Wild. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.461 ↗
42. Ronneberger, O., Fischer, P., and Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer.10.1007/978-3-319-24574-4_28 ↗
43. Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.
44. Google (2021, March 18). ARCore. Available online: https://developers.google.com/ar/discover/supported-devices.
45. Evaluation of Stopping Criterion in Contour Tracing Algorithms
46. A Delaunay Refinement Algorithm for Quality 2-Dimensional Mesh Generation10.1006/jagm.1995.1021 ↗
47. Automatic text-to-gesture rule generation for embodied conversational agents10.1002/cav.1944 ↗
48. An optical see-through display for mutual occlusion with a real-time stereovision system10.1016/S0097-8493(01)00119-4 ↗
49. Vorderer, P., Wirth, W., Gouveia, F.R., Biocca, F., Saari, T., Jäncke, F., Böcking, S., Schramm, H., Gysbers, A., and Hartmann, T. (2004). MEC spatial presence questionnaire (MEC-SPQ): Short documentation and instructions for application. Report to the European Community, Project Presence: MEC (IST-2001-37661), Available online: https://www.researchgate.net/publication/318531435_MEC_spatial_presence_questionnaire_MEC-SPQ_Short_documentation_and_instructions_for_application.
50. Kim, K., Bruder, G., and Welch, G. (2017, January 8–10). Exploring the effects of observed physicality conflicts on real-virtual human interaction in augmented reality. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology, Gothenburg, Sweden.10.1145/3139131.3139151 ↗
51. Cakmakci, O., Ha, Y., and Rolland, J. (2004, January 5). A Compact Optical See-Through Head-Worn Display with Occlusion Support. Proceedings of the Third IEEE and ACM International Symposium on Mixed and Augmented Reality, Arlington, VA, USA.
52. Norouzi, N., Kim, K., Lee, M., Schubert, R., Erickson, A., Bailenson, J., Bruder, G., and Welch, G. (2019, January 14–18). Walking your virtual dog: Analysis of awareness and proxemics with simulated support animals in augmented reality. Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2019, Beijing, China.10.1109/ISMAR.2019.000-8 ↗
53. Popovici, I., and Vatavu, R.D. (2019, January 14–18). Understanding Users’ Preferences for Augmented Reality Television. Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Beijing, China.10.1109/ISMAR.2019.00024 ↗
54. Haugstvedt, A.C., and Krogstie, J. (2012, January 5–8). Mobile augmented reality for cultural heritage: A technology acceptance study. Proceedings of the 2012 IEEE international symposium on mixed and augmented reality (ISMAR), Atlanta, GA, USA.10.1109/ISMAR.2012.6402563 ↗
55. Knierim, P., Woźniak, P.W., Abdelrahman, Y., and Schmidt, A. (2019, January 1–4). Exploring the potential of augmented reality in domestic environments. Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, Taipei, Taiwan.10.1145/3338286.3340142 ↗
56. Li, Y., Paluri, M., Rehg, J.M., and Dollár, P. (2016, January 27–30). Unsupervised learning of edges. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.179 ↗