Interactive Modelling in Augmented Reality with Subdivision Surfaces and Advanced User Gesture Recognition
Source abstract
Abstract
The paper discusses an integrated methodology to implement an interactive augmented reality 3D modelling environment with natural interaction, empowered by real-time gesture recognition. The methodology is developed from a geometry-sculpting algorithm based on the use of the subdivision surfaces approach to combine the ease and versatility of interactive modelling even of complex shapes, while maintaining high geometric continuity and smoothness. The interaction with the deformable elements of the geometry’s control cage to be divided uses an optimised version of the Grasp Active Feature/Object Active Feature algorithm developed from hand tracking and gesture recognition based on zero-invasive stereo-infrared techniques. Modelling, combined with an augmented reality environment, allows the modification of geometries having real objects as a reference and, in any case, a general spatial awareness during activities. The methodology was implemented and tested using an advanced mixed-reality headset, the Varjo XR-4, with hi-resolution pass-through and a second-generation Ultraleap for accurate and precise hand tracking.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Hand Gesture Recognition Systems · Augmented Reality Applications · Interactive and Immersive Displays
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Enterprise Engineering, University of Rome Tor Vergata, via del Politecnico 1, 00133 Rome, Italy
Department of Engineering, University Niccolò Cusano, via Don Carlo Gnocchi 3, 00166 Rome, Italy
Funding
Italian Ministry for Education and Research MIUR
References (46)
Show source-provided references
1. Future of industry 5.0 in society: Human-centric solutions, challenges and prospective research areas10.1186/s13677-022-00314-5 ↗
2. Pizoń, J., and Gola, A. (2023). Human–machine relationship—Perspective and future roadmap for industry 5.0 solutions. Machines, 11.10.3390/machines11020203 ↗
3. Industry 4.0 and Industry 5.0—Inception, conception and perception10.1016/j.jmsy.2021.10.006 ↗
4. Smart and user-centric manufacturing information recommendation using multimodal learning to support hu-man-robot collaboration in mixed reality environments10.1016/j.rcim.2024.102836 ↗
5. A mixed perception based human-robot collaborative maintenance approach driven by augmented reality and online deep reinforcement learning10.1016/j.rcim.2023.102568 ↗
6. Cellupica, A., Cirelli, M., Saggio, G., Gruppioni, E., and Valentini, P.P. (2024). An Interactive Digital-Twin Model for Virtual Reality Environments to Train in the Use of a Sensorized Upper-Limb Prosthesis. Algorithms, 17.10.3390/a17010035 ↗
7. A visual management and augmented-reality-based training module for the enhancement of short and long-term procedural knowledge retention in complex machinery setup10.1016/j.cie.2024.110478 ↗
8. A systematic review of augmented reality applications in maintenance10.1016/j.rcim.2017.06.002 ↗
9. BIM-based mixed reality application for bridge inspection10.1016/j.autcon.2024.105775 ↗
10. Impulse dynamics and augmented reality for real-time interactive digital twin exploration and interrogation10.1007/s12008-023-01704-y ↗
11. Automated positioning of 3D virtual scene in AR-based assembly and disassembly guiding system10.1007/s00170-014-6321-6 ↗
12. An auto-mated CAD-to-XR framework based on generative AI and Shrinkwrap modelling for a User-Centred design approach10.1016/j.aei.2024.102848 ↗
13. Development of an AR based method for augmentation of 3D CAD data onto a real ship block image
14. Exploration and evaluation of CAD modeling in virtual reality10.1080/16864360.2018.1462570 ↗
15. Multimodal virtual environments: Response times, attention, and presence10.1162/pres.15.5.515 ↗
16. Reipschläger, P., and Dachselt, R. (2019, January 10–13). Designar: Immersive 3D-modeling combining augmented reality with interactive displays. Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces, Daejeon, Republic of Korea.10.1145/3343055.3359718 ↗
17. 3D sketching in immersive environments: Shape from disordered ribbon strokes10.1016/j.cag.2024.103978 ↗
18. Rosales, E., Rodriguez, J., and Sheffer, A. (2019). SurfaceBrush: From virtual reality drawings to manifold surfaces. arXiv.10.1145/3306346.3322970 ↗
19. Arora, R., Kazi, R.H., Anderson, F., Grossman, T., Singh, K., and Fitzmaurice, G. (2017, January 6–11). Experimental Evaluation of Sketching on Surfaces in VR. Proceedings of the CHI’17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, USA.10.1145/3025453.3025474 ↗
20. Instant 3D design concept generation and visualization by real-time hand gesture recognition10.1016/j.compind.2013.04.012 ↗
21. Jiang, Y., Zhang, C., Fu, H., Cannavò, A., Lamberti, F., Lau, H.Y.K., and Wang, W. (2021, January 8–13). Handpainter-3D sketching in VR with hand-based physical proxy. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, Online Virtual.10.1145/3411764.3445302 ↗
22. Sculpting: An interactive volumetric modeling technique10.1145/127719.122747 ↗
23. Surface design using hand motion with smoothing
24. Zhu, X., and Yang, Y. (2024). Interactive Mesh Sculpting with Arbitrary Topologies in Head-Mounted VR Environments. Mathematics, 12.10.3390/math12152428 ↗
25. Jang, S.-A., Kim, H.-I., Woo, W., and Wakefield, G. (2014). Airsculpt: A wearable augmented reality 3D sculpting system. Distributed, Ambient, and Pervasive Interactions, Proceedings of the Second International Conference, DAPI 2014, Held as Part of HCI Interational 2014, Heraklion, Crete, Greece, 22–27 June 2014, Springer. Proceedings 2.10.1007/978-3-319-07788-8_13 ↗
26. Polygon mesh repairing: An application perspective10.1145/2431211.2431214 ↗
27. A novel haptics-based interface and sculpting system for physics-based geometric design
28. Interactive Sculpting Using Augmented-Reality, Mesh Morphing, and Force Feedback: Force-Feedback Capabilities in an Augmented Reality Environment10.1109/MCE.2017.2709598 ↗
29. Interactive virtual tools for manipulating NURBS surfaces in a virtual environment10.1115/1.1464132 ↗
30. VR–CAD integration: Multimodal immersive interaction and advanced haptic paradigms for implicit edition of CAD models
31. Butterworth, J., Davidson, A., Hench, S., and Olano, M.T. (April, January 29). 3DM: A three dimensional modeler using a head-mounted display. Proceedings of the 1992 Symposium on Interactive 3D Graphics, Cambridge, MA, USA.10.1145/147156.147182 ↗
32. Algorithms for reverse engineering boundary representation models10.1016/S0010-4485(01)00100-2 ↗
33. Ghali, S. (2008). Constructive solid geometry. Introduction to Geometric Computing, Springer.10.1007/978-1-84800-115-2 ↗
34. Malik, A., Lhachemi, H., and Shorten, R. (2023). A cyber-physical system to design 3D models using mixed reality technologies and deep learning for additive manufacturing. PLoS ONE, 18.10.1371/journal.pone.0289207 ↗
35. Visual interpretation of hand gestures for human-computer interaction: A review10.1109/34.598226 ↗
36. Vision based hand gesture recognition for human computer interaction: A survey10.1007/s10462-012-9356-9 ↗
37. Subdivision surfaces for CAD—An overview10.1016/j.cad.2004.08.008 ↗
38. Natural interface for interactive virtual assembly in augmented reality using Leap Motion Controller10.1007/s12008-018-0461-0 ↗
39. Natural interface in augmented reality interactive simulations10.1080/17452759.2012.682332 ↗
40. Interactive virtual assembling in augmented reality10.1007/s12008-009-0064-x ↗
41. Accuracy in fingertip tracking using Leap Motion Controller for interactive virtual applications10.1007/s12008-016-0339-y ↗
42. Catmull, E., and Clark, J. (1998). Recursively generated B-spline surfaces on arbitrary topological meshes. Seminal Graphics: Pioneering Efforts That Shaped the Field, ACM Digital Library.10.1145/280811.280992 ↗
43. Doo, D., and Sabin, M. (1998). Behaviour of recursive division surfaces near extraordinary points. Seminal Graphics: Pioneering Efforts That Shaped the Field, ACM Digital Library.10.1145/280811.280991 ↗
44. Loop, C. (1987). Smooth Subdivision Surfaces Based on Triangles. [Master’s Thesis, University of Utah].
45. Akenine-Moller, T., Haines, E., and Hoffman, N. (2019). Real-Time Rendering, AK Peters/CRC Press.10.1201/9781315365459 ↗
46. Zhang, H., and Hoff, K.E. (1997, January 27–30). Fast backface culling using normal masks. Proceedings of the 1997 Symposium on Interactive 3D Graphics, Providence, RI, USA.10.1145/253284.253314 ↗