Deep-Learning-Incorporated Augmented Reality Application for Engineering Lab Training
Source abstract
Abstract
Deep learning (DL) algorithms have achieved significantly high performance in object detection tasks. At the same time, augmented reality (AR) techniques are transforming the ways that we work and connect with people. With the increasing popularity of online and hybrid learning, we propose a new framework for improving students’ learning experiences with electrical engineering lab equipment by incorporating the abovementioned technologies. The DL powered automatic object detection component integrated into the AR application is designed to recognize equipment such as multimeter, oscilloscope, wave generator, and power supply. A deep neural network model, namely MobileNet-SSD v2, is implemented for equipment detection using TensorFlow’s object detection API. When a piece of equipment is detected, the corresponding AR-based tutorial will be displayed on the screen. The mean average precision (mAP) of the developed equipment detection model is 81.4%, while the average recall of the model is 85.3%. Furthermore, to demonstrate practical application of the proposed framework, we develop a multimeter tutorial where virtual models are superimposed on real multimeters. The tutorial includes images and web links as well to help users learn more effectively. The Unity3D game engine is used as the primary development tool for this tutorial to integrate DL and AR frameworks and create immersive scenarios. The proposed framework can be a useful foundation for AR and machine-learning-based frameworks for industrial and educational training.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Augmented Reality Applications · Advanced Neural Network Applications · QR Code Applications and Technologies
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Department of Electrical and Computer Engineering, Purdue University Northwest, Hammond, IN 46323, USA
Department of Applied Computing, Michigan Technological University, Houghton, MI 49931, USA
Funding
National Science Foundation
References (54)
Show source-provided references
1. A pilot study of the effectiveness of augmented reality to enhance the use of remote labs in electrical engineering education10.1007/s10956-011-9345-9 ↗
2. Evaluating the impact of the augmented reality learning environment on electronics laboratory skills of engineering students10.1002/cae.22156 ↗
3. Ray, S. (2019, January 14–16). A quick review of machine learning algorithms. Proceedings of the 2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon), Faridabad, India.10.1109/COMITCon.2019.8862451 ↗
4. Xue, M., and Zhu, C. (2009, January 25–26). A study and application on machine learning of artificial intelligence. Proceedings of the 2009 International Joint Conference on Artificial Intelligence, Hainan, China.10.1109/JCAI.2009.55 ↗
5. A framework for extended reality system development in manufacturing10.1109/ACCESS.2021.3056752 ↗
6. Nisiotis, L., and Alboul, L. (June, January 17). Work-In-Progress-An Intelligent Immersive Learning System Using AI, XR and Robots. Proceedings of the 2021 7th International Conference of the Immersive Learning Research Network (iLRN).10.23919/iLRN52045.2021.9459358 ↗
7. Software engineering for machine-learning applications: The road ahead10.1109/MS.2018.3571224 ↗
8. Hu, M., Weng, D., Chen, F., and Wang, Y. (2020, January 28–31). Object Detecting Augmented Reality System. Proceedings of the 2020 IEEE 20th International Conference on Communication Technology (ICCT), Nanning, China.10.1109/ICCT50939.2020.9295761 ↗
9. Ang, I.J.X., and Lim, K.H. (2019, January 28–30). Enhancing STEM Education Using Augmented Reality and Machine Learning. Proceedings of the 2019 7th International Conference on Smart Computing & Communications (ICSCC), Miri, Malaysia.10.1109/ICSCC.2019.8843619 ↗
10. Thiwanka, N., Chamodika, U., Priyankara, L., Sumathipala, S., and Weerasuriya, G. (2018, January 5–7). Augmented Reality Based Breadboard Circuit Building Guide Application. Proceedings of the 2018 3rd International Conference on Information Technology Research (ICITR), Moratuwa, Sri Lanka.10.1109/ICITR.2018.8736156 ↗
11. Sandoval Pérez, S., Gonzalez Lopez, J.M., Villa Barba, M.A., Jimenez Betancourt, R.O., Molinar Solís, J.E., Rosas Ornelas, J.L., Riberth García, G.I., and Rodriguez Haro, F. (2022). On the Use of Augmented Reality to Reinforce the Learning of Power Electronics for Beginners. Electronics, 11.10.3390/electronics11030302 ↗
12. Technologies, U. (2022, April 08). Unity Real-Time Development Platform. Available online: https://unity.com/.
13. Yu, H., Chen, C., Du, X., Li, Y., Rashwan, A., Hou, L., Jin, P., Yang, F., Liu, F., and Kim, J. (2022, April 28). TensorFlow 1 Detection Model Zoo. Available online: https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md/.
14. Big data deep learning: Challenges and perspectives10.1109/ACCESS.2014.2325029 ↗
15. Alom, M.Z., Taha, T.M., Yakopcic, C., Westberg, S., Sidike, P., Nasrin, M.S., Hasan, M., Van Essen, B.C., Awwal, A.A., and Asari, V.K. (2019). A state-of-the-art survey on deep learning theory and architectures. Electronics, 8.10.3390/electronics8030292 ↗
16. Deng, L., and Liu, Y. (2018). Deep Learning in Natural Language Processing, Springer.10.1007/978-981-10-5209-5 ↗
17. Deng, L., Hinton, G., and Kingsbury, B. (2013, January 26–31). New types of deep neural network learning for speech recognition and related applications: An overview. Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, Vancouve, BC, Canada.10.1109/ICASSP.2013.6639344 ↗
18. Nishani, E., and Çiço, B. (2017, January 11–15). Computer vision approaches based on deep learning and neural networks: Deep neural networks for video analysis of human pose estimation. Proceedings of the 2017 6th Mediterranean Conference on Embedded Computing (MECO), Bar, Montenegro.10.1109/MECO.2017.7977207 ↗
19. Gradient-based learning applied to document recognition10.1109/5.726791 ↗
20. Pandiya, M., Dassani, S., and Mangalraj, P. (2020, January 11–12). Analysis of Deep Learning Architectures for Object Detection-A Critical Review. Proceedings of the 2020 IEEE-HYDCON, Hyderabad, India.10.1109/HYDCON48903.2020.9242776 ↗
21. Arora, D., Garg, M., and Gupta, M. (2020, January 18–19). Diving deep in deep convolutional neural network. Proceedings of the 2020 2nd International Conference on Advances in Computing, Communication Control and Networking (ICACCCN), Greater Noida, India.10.1109/ICACCCN51052.2020.9362907 ↗
22. Region-based convolutional networks for accurate object detection and segmentation10.1109/TPAMI.2015.2437384 ↗
23. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (2016, January 27–30). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.10.1109/CVPR.2016.91 ↗
24. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). SSD: Single Shot MultiBox Detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
25. Ryu, J., and Kim, S. (2018, January 17–20). Chinese character detection using modified single shot multibox detector. Proceedings of the 2018 18th International Conference on Control, Automation and Systems (ICCAS), PyeongChang, Korea.
26. Chiu, Y.C., Tsai, C.Y., Ruan, M.D., Shen, G.Y., and Lee, T.T. (September, January 31). Mobilenet-SSDv2: An improved object detection model for embedded systems. Proceedings of the 2020 International Conference on System Science and Engineering (ICSSE), Kagawa, Japan.10.1109/ICSSE50014.2020.9219319 ↗
27. Heirman, J., Selleri, S., De Vleeschauwer, T., Hamesse, C., Bellemans, M., Schoofs, E., and Haelterman, R. (2020, January 14–18). Exploring the possibilities of Extended Reality in the world of firefighting. Proceedings of the 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), Utrecht, The Netherlands.10.1109/AIVR50618.2020.00055 ↗
28. Andrade, T.M., Smith-Creasey, M., and Roscoe, J.F. (2020, January 9–10). Discerning User Activity in Extended Reality Through Side-Channel Accelerometer Observations. Proceedings of the 2020 IEEE International Conference on Intelligence and Security Informatics (ISI), Arlington, VA, USA.10.1109/ISI49825.2020.9280516 ↗
29. Dandachi, G., Assoum, A., Elhassan, B., and Dornaika, F. (May, January 29). Machine learning schemes in augmented reality for features detection. Proceedings of the 2015 Fifth International Conference on Digital Information and Communication Technology and Its Applications (DICTAP), Beirut, Lebanon.10.1109/DICTAP.2015.7113179 ↗
30. Sendari, S., Anggreani, D., Jiono, M., Nurhandayani, A., and Suardi, C. (2020, January 19). Augmented reality performance in detecting hardware components using marker based tracking method. Proceedings of the 2020 4th International Conference on Vocational Education and Training (ICOVET), Malang, Indonesia.10.1109/ICOVET50258.2020.9229895 ↗
31. Mahurkar, S. (2018, January 8–10). Integrating YOLO Object Detection with Augmented Reality for iOS Apps. Proceedings of the 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York, NY, USA.10.1109/UEMCON.2018.8796579 ↗
32. Augmented reality types and popular use cases
33. Poetker, B. (2022, April 08). What Is Augmented Reality? (+Most Common Types of AR Used Today). Available online: https://www.g2.com/articles/augmented-reality.
34. Gao, Y.F., Wang, H.Y., and Bian, X.N. (2016, January 10–13). Marker tracking for video-based augmented reality. Proceedings of the 2016 International Conference on Machine Learning and Cybernetics (ICMLC), Jeju Island, Korea.10.1109/ICMLC.2016.7873011 ↗
35. Sendari, S., and Firmansah, A. (2020, January 19). Performance analysis of augmented reality based on vuforia using 3d marker detection. Proceedings of the 2020 4th International Conference on Vocational Education and Training (ICOVET), Malang, Indonesia.10.1109/ICOVET50258.2020.9230276 ↗
36. Vidya, K., Deryl, R., Dinesh, K., Rajabommannan, S., and Sujitha, G. (2014, January 5–7). Enhancing hand interaction patterns for virtual objects in mobile augmented reality using marker-less tracking. Proceedings of the 2014 International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India.10.1109/IndiaCom.2014.6828052 ↗
37. Beier, D., Billert, R., Bruderlin, B., Stichling, D., and Kleinjohann, B. (2003, January 7–10). Marker-less vision based tracking for mobile augmented reality. Proceedings of the The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, Tokyo, Japan.
38. Pooja, J., Vinay, M., Pai, V.G., and Anuradha, M. (2020, January 6–8). Comparative analysis of marker and marker-less augmented reality in education. Proceedings of the 2020 IEEE International Conference for Innovation in Technology (INOCON), Bangalore, India.
39. Batuwanthudawa, B., and Jayasena, K. (2020, January 10–11). Real-Time Location based Augmented Reality Advertising Platform. Proceedings of the 2020 2nd International Conference on Advancements in Computing (ICAC), Malabe, Sri Lanka.10.1109/ICAC51239.2020.9357261 ↗
40. Unal, M., Bostanci, E., Sertalp, E., Guzel, M.S., and Kanwal, N. (2018, January 19–21). Geo-location based augmented reality application for cultural heritage using drones. Proceedings of the 2018 2nd International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT), Ankara, Turkey.10.1109/ISMSIT.2018.8567073 ↗
41. Dynamic superimposition of synthetic objects on rigid and simple-deformable real objects10.1016/S0097-8493(02)00180-2 ↗
42. Ketchell, S., Chinthammit, W., and Engelke, U. (2019, January 14–16). Situated storytelling with SLAM enabled augmented reality. Proceedings of the The 17th International Conference on Virtual-Reality Continuum and Its Applications in Industry, Brisbane, QLD, Australia.10.1145/3359997.3365681 ↗
43. Knopp, S., Klimant, P., Schaffrath, R., Voigt, E., Fritzsche, R., and Allmacher, C. (2019, January 10–18). Hololens ar-using vuforia-based marker tracking together with text recognition in an assembly scenario. Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Beijing, China.10.1109/ISMAR-Adjunct.2019.00030 ↗
44. Soulami, K.B., Ghribi, E., Labyed, Y., Saidi, M.N., Tamtaoui, A., and Kaabouch, N. (2019, January 20–22). Mixed-reality aided system for glioblastoma resection surgery using microsoft HoloLens. Proceedings of the 2019 IEEE International Conference on Electro Information Technology (EIT), Brookings, SD, USA.10.1109/EIT.2019.8833738 ↗
45. Lee, J.D., Wu, H.K., and Wu, C.T. (2018, January 9–12). A projection-based AR system to display brain angiography via stereo vision. Proceedings of the 2018 IEEE 7th Global Conference on Consumer Electronics (GCCE), Nara, Japan.10.1109/GCCE.2018.8574761 ↗
46. (2022, April 08). Vuforia Developer Library. Introduction to Model Targets. Available online: https://library.vuforia.com/articles/Solution/introduction-model-targets-unity.html.
47. Zhang, S., Tian, J., Zhai, X., and Ji, Z. (2020, January 6–8). Detection of Porcine Huddling Behaviour Based on Improved Multi-view SSD. Proceedings of the 2020 Chinese Automation Congress (CAC), Shanghai, China.10.1109/CAC51589.2020.9327887 ↗
48. Rios, A.C., dos Reis, D.H., da Silva, R.M., Cuadros, M.A.D.S.L., and Gamarra, D.F.T. (2021, January 15–18). Comparison of the YOLOv3 and SSD MobileNet v2 Algorithms for Identifying Objects in Images from an Indoor Robotics Dataset. Proceedings of the 2021 14th IEEE International Conference on Industry Applications (INDUSCON), Sao Paulo, Brazil.10.1109/INDUSCON51756.2021.9529585 ↗
49. Phadnis, R., Mishra, J., and Bendale, S. (2018, January 20–21). Objects talk-object detection and pattern tracking using tensorflow. Proceedings of the 2018 Second International Conference on Inventive Communication and Computational Technologies (ICICCT), Coimbatore, India.10.1109/ICICCT.2018.8473331 ↗
50. Lin, T. (2022, April 08). LabelImg. Available online: https://github.com/tzutalin/labelImg.
51. Kilic, I., and Aydin, G. (2020, January 25–27). Traffic sign detection and recognition using tensorflow’s object detection API with a new benchmark dataset. Proceedings of the 2020 International Conference on Electrical Engineering (ICEE), Istanbul, Turkey.10.1109/ICEE49691.2020.9249914 ↗
52. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the 28th Annual Conference on Neural Information Processing Systems, Montreal, Canada.
53. An Efficient Method of Scanning and Tracking for AR
54. (2022, April 08). Enoxsoftware. About OpenCV for Unity. Available online: https://enoxsoftware.com/opencvforunity/documentation/about-opencv-for-unity.