Multimodal Natural Human–Computer Interfaces for Computer-Aided Design: A Review Paper
Source abstract
Abstract
Computer-aided design (CAD) systems have advanced to become a critical tool in product design. Nevertheless, they still primarily rely on the traditional mouse and keyboard interface. This limits the naturalness and intuitiveness of the 3D modeling process. Recently, a multimodal human–computer interface (HCI) has been proposed as the next-generation interaction paradigm. Widening the use of a multimodal HCI provides new opportunities for realizing natural interactions in 3D modeling. In this study, we conducted a literature review of a multimodal HCI for CAD to summarize the state-of-the-art research and establish a solid foundation for future research. We explore and categorize the requirements for natural HCIs and discuss paradigms for their implementation in CAD. Following this, factors to evaluate the system performance and user experience of a natural HCI are summarized and analyzed. We conclude by discussing challenges and key research directions for a natural HCI in product design to inspire future studies.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Tactile and Sensory Interactions · Gaze Tracking and Assistive Technology · Interactive and Immersive Displays
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Institute of Industrial and Intelligent Systems Engineering, Beijing Institute of Technology, Beijing 100081, China
Brain and Cognition Research Unit, KU Leuven, 3000 Leuven, Belgium
Center for Cognitive Science, University of Kaiserslautern, 67663 Kaiserslautern, Germany
Centro de Investigación Nebrija en Cognición, Universidad Nebrija, 28015 Madrid, Spain
Funding
National Ministry Projects of China
References (100+)
Show source-provided references
1. Allen, J., and Kouppas, P. (2012). Computer Aided Design: Past, Present, Future. Design and Designing: A Critical Introduction, BERG.
2. A Review of Computer-Aided Design/Computer-Aided Manufacture Techniques for Removable Denture Fabrication10.4103/1305-7456.178304 ↗
3. Intuitive 3D Computer-Aided Design (CAD) System with Multimodal Interfaces
4. Wu, K.C., and Fernando, T. (2004, January 10–13). Novel Interface for Future CAD Environments. Proceedings of the 2004 IEEE International Conference on Systems, Man and Cybernetics, The Hague, The Netherlands.
5. Conceptual Three-Dimensional Modeling Using Intuitive Gesture-Based Midair Three-Dimensional Sketching Technique10.1115/1.4040982 ↗
6. GG Interaction: A Gaze–Grasp Pose Interaction for 3D Virtual Object Selection10.1007/s12193-019-00305-y ↗
7. A Novel User-Based Gesture Vocabulary for Conceptual Design10.1016/j.ijhcs.2021.102609 ↗
8. Friedrich, M., Langer, S., and Frey, F. (2021, January 8–10). Combining Gesture and Voice Control for Mid-Air Manipulation of CAD Models in VR Environments. Proceedings of the VISIGRAPP 2021—16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Online.10.5220/0010170501190127 ↗
9. Brain-Computer Interfaces (BCI) Based 3D Computer-Aided Design (CAD): To Improve the Efficiency of 3D Modeling for New Users
10. Multi-Modal Interaction for 3D Modeling
11. Khan, S., Tunçer, B., Subramanian, R., and Blessing, L. (2019, January 26–28). 3D CAD Modeling Using Gestures and Speech: Investigating CAD Legacy and Non-Legacy Procedures. Proceedings of the “Hello, Culture”—18th International Conference on Computer-Aided Architectural Design Future (CAAD Future 2019), Daejeon, Korea.
12. Gesture and Speech Elicitation for 3D CAD Modeling in Conceptual Design10.1016/j.autcon.2019.102847 ↗
13. Natural User Interfaces
14. Shao, L., Shan, C., Luo, J., and Etoh, M. (2010). Multimedia Interaction and Intelligent User Interfaces: Principles, Methods and Applications, Springer Science & Business Media.10.1007/978-1-84996-507-1 ↗
15. Chu, M., and Begole, B. (2010). Natural and Implicit Information-Seeking Cues in Responsive Technology. Human-Centric Interfaces for Ambient Intelligence, Elsevier.10.1016/B978-0-12-374708-2.00017-6 ↗
16. Human-Computer Interaction: Overview on State of the Art
17. The Future of Integrated CAD/CAM Systems: The Boeing Perspective10.1109/MCG.1982.1674105 ↗
18. Fetter, W. (1978, January 19–21). A Computer Graphic Human Figure System Applicable to Kineseology. Proceedings of the 15th Design Automation Conference, Las Vegas, GA, USA.10.1109/DAC.1978.1585188 ↗
19. CyberCAD: A Collaborative Approach in 3D-CAD Technology in a Multimedia-Supported Environment10.1016/S0166-3615(03)00100-3 ↗
20. Tornincasa, S., and Di Monaco, F. (2010, January 11–18). The Future and the Evolution of CAD. Proceedings of the 14th International Research/Expert Conference: Trends in the Development of Machinery and Associated Technology, Mediterranean Cruise.
21. A Review of Dental CAD/CAM: Current Status and Future Perspectives from 20 Years of Experience10.4012/dmj.28.44 ↗
22. The Integration of CAD/CAM and Rapid Prototyping in Product Development: A Review10.1016/j.matpr.2015.07.319 ↗
23. Lichten, L. (1984, January 8–14). The Emerging Technology of CAD/CAM. Proceedings of the 1984 Annual Conference of the ACM on The Fifth Generation Challenge, ACM ’84, San Francisco, CA, USA.10.1145/800171.809635 ↗
24. Knowledge-Guided Inference for Voice-Enabled CAD
25. Thakur, A., and Rai, R. (2015, January 2–5). User Study of Hand Gestures for Gesture Based 3D CAD Modeling. Proceedings of the ASME Design Engineering Technical Conference, Boston, MA, USA.10.1115/DETC2015-46086 ↗
26. Baig, M.Z., and Kavakli, M. (2019, January 18–20). Analyzing Novice and Expert User’s Cognitive Load in Using a Multi-Modal Interface System. Proceedings of the 2018 26th International Conference on Systems Engineering (ICSEng), Sydney, Australia.10.1109/ICSENG.2018.8638206 ↗
27. Esquivel, J.C.R., Viveros, A.M., and Perry, N. (2014). Gestures for Interaction between the Software CATIA and the Human via Microsoft Kinect. International Conference on Human-Computer Interaction, Springer.
28. Human Factors Study on the Usage of BCI Headset for 3D CAD Modeling
29. Multimodal Interaction: A Review10.1016/j.patrec.2013.07.003 ↗
30. Bhat, R., Deshpande, A., Rai, R., and Esfahani, E.T. (2013). BCI-Touch Based System: A Multimodal CAD Interface for Object Manipulation. Volume 12: Systems and Design, Proceedings of the ASME 2013 International Mechanical Engineering Congress and Exposition, San Diego, CA, USA, 15–21 November 2013, American Society of Mechanical Engineers.10.1115/IMECE2013-65648 ↗
31. User-Centered Modeling and Evaluation of Multimodal Interfaces10.1109/JPROC.2003.817127 ↗
32. Multi-Modal User Interaction Method Based on Gaze Tracking and Gesture Recognition10.1016/j.image.2012.10.007 ↗
33. Multimodal Human Computer Interaction: A Survey
34. GaFinC: Gaze and Finger Control Interface for 3D Model Manipulation in CAD Application10.1016/j.cad.2013.08.039 ↗
35. A Comparative Evaluation of Human Interaction for Design and Assembly of 3D CAD Models in Desktop and Immersive Environments10.1007/s12008-012-0144-1 ↗
36. Computers, Words and Pictures10.1016/S0142-694X(97)85459-2 ↗
37. Cognitive, Collaborative, Conceptual and Creative—Four Characteristics of the next Generation of Knowledge-Based CAD Systems: A Study in Biologically Inspired Design10.1016/j.cad.2011.03.010 ↗
38. Impact of CAD Tools on Creative Problem Solving in Engineering Design
39. Evaluating an Eye Tracking Interface for a Two-Dimensional Sketch Editor
40. Yoon, S.M., and Graf, H. (2008, January 27–31). Eye Tracking Based Interaction with 3d Reconstructed Objects. In Proceeding of the 16th ACM International Conference on Multimedia—MM ’08, Vancouver, BC, Canada.10.1145/1459359.1459501 ↗
41. A Survey of 3D Object Selection Techniques for Virtual Environments10.1016/j.cag.2012.12.003 ↗
42. Gesture Interface for 3d Cad Modeling Using Kinect10.3722/cadaps.2013.663-669 ↗
43. Zhong, K., Kang, J., Qin, S., and Wang, H. (2011, January 18–20). Rapid 3D Conceptual Design Based on Hand Gesture. Proceedings of the 2011 3rd International Conference on Advanced Computer Control, Harbin, China.10.1109/ICACC.2011.6016395 ↗
44. Xiao, Y., and Peng, Q. (2017, January 21–25). A Hand Gesture-Based Interface for Design Review Using Leap Motion Controller. Proceedings of the 21st International Conference on Engineering Design (ICED 17), Vancouver, BC, Canada.
45. Shape-It-Up: Hand Gesture Based Creative Expression of 3D Shapes Using Intelligent Generalized Cylinders10.1016/j.cad.2012.10.011 ↗
46. Vinayak, and Ramani, K (2015). A Gesture-Free Geometric Approach for Mid-Air Expression of Design Intent in 3D Virtual Pottery. CAD Comput. Aided Des., 69, 11–24.10.1016/j.cad.2015.06.006 ↗
47. Huang, J., and Rai, R. (2014, January 17–20). Hand Gesture Based Intuitive CAD Interface. Proceedings of the ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Buffalo, NY, USA.10.1115/DETC2014-34070 ↗
48. Kou, X.Y., Liu, X.C., and Tan, S.T. (2009, January 1–13). Quadtree Based Mouse Trajectory Analysis for Efficacy Evaluation of Voice-Enabled CAD. Proceedings of the 2009 IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurements Systems, Hong Kong, China.10.1109/VECIMS.2009.5068892 ↗
49. Samad, T., and Director, S.W. (1985, January 23–26). Towards a Natural Language Interface for CAD. Proceedings of the 22nd ACM/IEEE Design Automation Conference, Las Vegas, NV, USA.10.1109/DAC.1985.1585905 ↗
50. Talk and Draw: Bundling Speech and Graphics10.1109/2.56872 ↗
51. Design by Talking with Computers10.3722/cadaps.2008.266-277 ↗
52. A Framework for Speech-Oriented CAD and BIM Systems10.1007/978-3-662-47386-3_18 ↗
53. Behera, A.K., and McKay, A. (September, January 28). Designs That Talk and Listen: Integrating Functional Information Using Voice-Enabled CAD Systems. Proceedings of the 25th European Signal Processing Conference, Kos, Greek.
54. Natural Voice-Enabled CAD: Modeling via Natural Discourse10.3722/cadaps.2009.125-136 ↗
55. A Voice-Based Annotation System for Collaborative Computer-Aided Design
56. Brain-Computer Interfaces for 1-D and 2-D Cursor Control: Designs Using Volitional Control of the EEG Spectrum or Steady-State Visual Evoked Potentials10.1109/TNSRE.2006.875578 ↗
57. Esfahani, E.T., and Sundararajan, V. (2011, January 28–31). Using Brain Computer Interfaces for Geometry Selection in CAD Systems: P300 Detection Approach. Proceedings of the 31st Computers and Information in Engineering Conference, Parts A and B, Washington, DC, USA.10.1115/DETC2011-48775 ↗
58. Classification of Primitive Shapes Using Brain-Computer Interfaces10.1016/j.cad.2011.04.008 ↗
59. Huang, Y.C., Chen, K.L., Wu, M.Y., Tu, Y.W., and Huang, S.C.C. (2015, January 2–7). Brain-Computer Interface Approach to Computer-Aided Design: Rotate and Zoom in/out in 3ds Max via Imagination. Proceedings of the International Conferences Interfaces and Human Computer Interaction 2015 (IHCI 2015), Los Angeles, CA, USA. Game and Entertainment Technologies 2015 (GET 2015), Las Palmas de Gran Canaria, Spain, 22–24 July 2015, Computer Graphics, Visualization, Computer Vision and Image Processing 2015 (CGVCVIP 2015), Las Palmas de Gran Canaria, Spain, 22–24 July 2015.
60. Postelnicu, C., Duguleana, M., Garbacia, F., and Talaba, D. (2014, January 8–10). Towards P300 Based Brain Computer Interface for Computer Aided Design. Proceedings of the 11th EuroVR 2014, Bremen, Germany.
61. Verma, A., and Rai, R. (2013, January 4–7). Creating by Imagining: Use of Natural and Intuitive BCI in 3D CAD Modeling. Proceedings of the 33rd Computers and Information in Engineering Conference, Portland, OR, USA.
62. Moustakas, K., and Tzovaras, D. (2005, January 12–18). MASTER-PIECE: A Multimodal (Gesture + Speech) Interface for 3D Model Search and Retrieval Integrated in a Virtual Assembly Application. Proceedings of the eINTERFACE′05-Summer Workshop on Multimodal Interfaces, Mons, Belgium.
63. Bolt, R.A. (1980, January 14–18). Put-That-There. Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques—SIGGRAPH ’80, Seattle, WA, USA.10.1145/800250.807503 ↗
64. Khan, S., Rajapakse, H., Zhang, H., Nanayakkara, S., Tuncer, B., and Blessing, L. (December, January 28). GesCAD: An Intuitive Interface for Conceptual Architectural Design. Proceedings of the 29th Australian Conference on Computer-Human Interaction, Brisbane, Australia.
65. Speech and Gestures for Graphic Image Manipulation10.1145/67450.67496 ↗
66. Arangarasan, R., and Gadh, R. (2000, January 10–13). Geometric Modeling and Collaborative Design in a Multi-Modal Multi-Sensory Virtual Environment. Proceedings of the ASME 2000 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Baltimore, MD, USA.10.1115/DETC2000/CIE-14592 ↗
67. Multi-Sensory User Interface for a Virtual-Reality-Based Computeraided Design System
68. Chu, C.-C.P., Dani, T.H., and Gadh, R. (1997, January 20–25). Multimodal Interface for a Virtual Reality Based Computer Aided Design System. Proceedings of the International Conference on Robotics and Automation, Albuquerque, NM, USA.
69. Pouke, M., Karhu, A., Hickey, S., and Arhippainen, L. (2012, January 26–30). Gaze Tracking and Non-Touch Gesture Based Interaction Method for Mobile 3D Virtual Spaces. Proceedings of the 24th Australian Computer-Human Interaction Conference on OzCHI ’12, Melbourne, Australia.10.1145/2414536.2414614 ↗
70. Shafiei, S.B., and Esfahani, E.T. (2014, January 17–20). Aligning Brain Activity and Sketch in Multi-Modal CAD Interface. Proceedings of the ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Buffalo, NY, USA.10.1115/DETC2014-34816 ↗
71. Sharma, A., and Madhvanath, S. (2011, January 14–18). MozArt: A Multimodal Interface for Conceptual 3D Modeling. Proceedings of the 13th International Conference on Multimodal Interfaces, Alicante, Spain.10.1145/2070481.2070538 ↗
72. VR–CAD Integration: Multimodal Immersive Interaction and Advanced Haptic Paradigms for Implicit Edition of CAD Models
73. Mogan, G., Talaba, D., Girbacia, F., Butnaru, T., Sisca, S., and Aron, C. (2008, January 6–8). A Generic Multimodal Interface for Design and Manufacturing Applications. Proceedings of the 2nd International Workshop Virtual Manufacturing (VirMan08), Torino, Italy.
74. Towards Hybrid Modelling Environments—Merging Desktop-CAD and Virtual Reality-Technologies10.1016/j.cirp.2010.03.102 ↗
75. Ren, X., Zhang, G., and Dai, G. (2000, January 14–16). An Experimental Study of Input Modes for Multimodal Human-Computer Interaction. Proceedings of the International Conference on Multimodal Interfaces, Beijing, China.10.1007/3-540-40063-X_7 ↗
76. Multimodal Human–Computer Interaction: A Survey10.1016/j.cviu.2006.10.019 ↗
77. Eye-Gaze Computer Interfaces: Computers That Sense Eye Position on the Display10.1109/MC.1993.620436 ↗
78. Sharma, C., and Dubey, S.K. (2014, January 5–7). Analysis of Eye Tracking Techniques in Usability and HCI Perspective. Proceedings of the 2014 International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India.10.1109/IndiaCom.2014.6828034 ↗
79. The Development of Gaze Following and Its Relation to Language10.1111/j.1467-7687.2005.00445.x ↗
80. A Breadth-First Survey of Eye-Tracking Applications10.3758/BF03195475 ↗
81. Kumar, M., Paepcke, A., and Winograd, T. (May, January 28). Eyepoint: Practical Pointing and Selection Using Gaze and Keyboard. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, San Jose, CA, USA.
82. Majaranta, P., and Räihä, K.-J. (2002, January 25–27). Twenty Years of Eye Typing: Systems and Design Issues. Proceedings of the 2002 Symposium on Eye Tracking Research & Applications, New Orleans, LA, USA.10.1145/507075.507076 ↗
83. Wobbrock, J.O., Rubinstein, J., Sawyer, M.W., and Duchowski, A.T. (2008, January 26–28). Longitudinal Evaluation of Discrete Consecutive Gaze Gestures for Text Entry. Proceedings of the 2008 Symposium on Eye Tracking Research & Applications, Savannah, GA, USA.10.1145/1344471.1344475 ↗
84. Hornof, A.J., and Cavender, A. (2005, January 2–7). EyeDraw: Enabling Children with Severe Motor Impairments to Draw with Their Eyes. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Portland, OR, USA.10.1145/1054972.1054995 ↗
85. HCI and Eye Tracking Technology for Learning Effect10.1016/j.sbspro.2012.11.073 ↗
86. Remote Vocational Learning Opportunities—A Comparative Eye-Tracking Investigation of Educational 2D Videos versus 360° Videos for Car Mechanics10.1111/bjet.13162 ↗
87. Dickie, C., Vertegaal, R., Sohn, C., and Cheng, D. (2005, January 23–26). EyeLook: Using Attention to Facilitate Mobile Media Consumption. Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, Seattle, WA, USA.10.1145/1095034.1095050 ↗
88. Nagamatsu, T., Yamamoto, M., and Sato, H. (2010, January 10–15). MobiGaze: Development of a Gaze Interface for Handheld Mobile Devices. Proceedings of the Human Factors in Computing Systems, New York, NY, USA.10.1145/1753846.1753983 ↗
89. Miluzzo, E., Wang, T., and Campbell, A.T. (2010, January 30). Eyephone: Activating Mobile Phones with Your Eyes. Proceedings of the Second ACM SIGCOMM Workshop on Networking, Systems, and Applications on Mobile Handhelds, New Delhi, India.10.1145/1851322.1851328 ↗
90. Franslin, N.M.F., and Ng, G.W. (2021, January 28–29). Vision-Based Dynamic Hand Gesture Recognition Techniques and Applications: A Review. Proceedings of the 8th International Conference on Computational Science and Technology, Labuan, Malaysia.
91. Hand Gestures in CAD Systems
92. Design Review of Cad Models Using a NUI Leap Motion Sensor
93. Kaur, H., and Rani, J. (2016, January 4–6). A Review: Study of Various Techniques of Hand Gesture Recognition. Proceedings of the 1st IEEE International Conference on Power Electronics, Intelligent Control and Energy Systems, ICPEICES 2016, Delhi, India.10.1109/ICPEICES.2016.7853514 ↗
94. Conceptual Design and Modification of Freeform Surfaces Using Dual Shape Representations in Augmented Reality Environments10.1016/j.cad.2011.05.009 ↗
95. Pareek, S., Sharma, V., and Esfahani, E.T. (2015, January 2–5). Human Factor Study in Gesture Based Cad Environment. Proceedings of the International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Boston, MA, USA.10.1115/DETC2015-47707 ↗
96. Gesture-Based System for next Generation Natural and Intuitive Interfaces10.1017/S0890060418000045 ↗
97. The State of Speech in HCI: Trends, Themes and Challenges10.1093/iwc/iwz016 ↗
98. Approach to Solid Modeling in a Semi-Immersive Virtual Environment10.1016/S0097-8493(99)00154-5 ↗
99. Rezeika, A., Benda, M., Stawicki, P., Gembler, F., Saboor, A., and Volosyak, I. (2018). Brain–Computer Interface Spellers: A Review. Brain Sci., 8.10.3390/brainsci8040057 ↗
100. A Review of Hybrid Brain-Computer Interface Systems10.1155/2013/187024 ↗