Towards Digital Twins of 3D Reconstructed Apparel Models with an End-to-End Mobile Visualization
Source abstract
Abstract
Digital twin technologies are still developing and are being increasingly leveraged to facilitate daily life activities. This study presents a novel approach for leveraging the capability of mobile devices for photo collection, cloud processing, and deep learning-based 3D generation, with seamless display in virtual reality (VR) wearables. The purpose of our study is to provide a system that makes use of cloud computing resources to offload the resource-intensive activities of 3D reconstruction and deep-learning-based scene interpretation. We establish an end-to-end pipeline from 2D to 3D reconstruction, which automatically builds accurate 3D models from collected photographs using sophisticated deep-learning techniques. These models are then converted to a VR-compatible format, allowing for immersive and interactive experiences on wearable devices. Our findings attest to the completion of 3D entities regenerated by the CAP–UDF model using ShapeNetCars and Deep Fashion 3D datasets with a discrepancy in L2 Chamfer distance of only 0.089 and 0.129, respectively. Furthermore, the demonstration of the end-to-end process from 2D capture to 3D visualization on VR occurs continuously.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: 3D Shape Modeling and Analysis · Virtual Reality Applications and Impacts · Anatomy and Medical Technology
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
International Graduate Program of Electrical Engineering and Computer Science, National Taipei University of Technology, Taipei 10608, Taiwan
Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei 106, Taiwan
Department of Computer Science and Information Engineering, National Taipei University of Technology, Taipei 10608, Taiwan
Funding
Ministry of Science and Technology
References (55)
Show source-provided references
1. Grieves, M., and Vickers, J. (2017). Transdisciplinary Perspectives on Complex Systems: New Findings and Approaches, Springer International Publishing.
2. Digital Twin: Generalization, characterization and implementation10.1016/j.dss.2021.113524 ↗
3. Social Cyber-Physical System\s and Digital Twins Networks: A perspective about the future digital twin ecosystems10.1016/j.ifacol.2022.08.006 ↗
4. Review of digital twin about concepts, technologies, and industrial applications10.1016/j.jmsy.2020.06.017 ↗
5. Collins, J., Goel, S., Deng, K., Luthra, A., Xu, L., Gundogdu, E., Zhang, X., Vicente, T.F.Y., Dideriksen, T., and Arora, H. (2022, January 18–24). Abo: Dataset and benchmarks for real-world 3d object understanding. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.02045 ↗
6. An out-of-core method for GPU image mapping on large 3D scenarios of the real world10.1016/j.future.2022.03.022 ↗
7. The current status on 3D scanning and CAD/CAM applications in textile research
8. A case study on use of 3D scanning for reverse engineering and quality control10.1016/j.matpr.2021.01.828 ↗
9. Son, K., and Lee, K.B. (2020). Effect of tooth types on the accuracy of dental 3d scanners: An in vitro study. Materials, 13.10.3390/ma13071744 ↗
10. Deep learning for 3d point clouds: A survey10.1109/TPAMI.2020.3005434 ↗
11. Neural unsigned distance fields for implicit function learning
12. Venkatesh, R., Karmali, T., Sharma, S., Ghosh, A., Babu, R.V., Jeni, L.A., and Singh, M. (2021, January 11–17). Deep implicit surface point prediction networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01242 ↗
13. Zhao, F., Wang, W., Liao, S., and Shao, L. (2021, January 11–17). Learning anchored unsigned distance functions with gradient direction alignment for single-view garment reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada.10.1109/ICCV48922.2021.01244 ↗
14. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions10.1186/s40537-021-00444-8 ↗
15. Phanichraksaphong, V., and Tsai, W.H. (2023). Automatic Assessment of Piano Performances Using Timbre and Pitch Features. Electronics, 12.10.3390/electronics12081791 ↗
16. Liu, J., Ji, P., Bansal, N., Cai, C., Yan, Q., Huang, X., and Xu, Y. (2022, January 18–24). Planemvs: 3d plane reconstruction from multi-view stereo. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.00847 ↗
17. Cernica, D., Benedek, I., Polexa, S., Tolescu, C., and Benedek, T. (2021). 3D Printing—A Cutting Edge Technology for Treating Post-Infarction Patients. Life, 11.10.3390/life11090910 ↗
18. Human digital twin for fitness management10.1109/ACCESS.2020.2971576 ↗
19. Is human digital twin possible?10.1016/j.cmpbup.2021.100014 ↗
20. Li, X., Cao, J., Liu, Z., and Luo, X. (2021). Sustainable business model based on digital twin platform network: The inspiration from Haier’s case study in China. Sustainability, 12.10.3390/su12030936 ↗
21. Al-Ali, A.R., Gupta, R., Zaman Batool, T., Landolsi, T., Aloul, F., and Al Nabulsi, A. (2020). Digital twin conceptual model within the context of internet of things. Future Internet, 12.10.3390/fi12100163 ↗
22. Štroner, M., Křemen, T., and Urban, R. (2022). Progressive Dilution of Point Clouds Considering the Local Relief for Creation and Storage of Digital Twins of Cultural Heritage. Appl. Sci., 12.10.3390/app122211540 ↗
23. Niccolucci, F., Felicetti, A., and Hermon, S. (2022). Populating the Data Space for Cultural Heritage with Heritage Digital Twins. Data, 7.10.3390/data7080105 ↗
24. Lv, Z., Shang, W.L., and Guizani, M. (2022). Impact of Digital Twins and Metaverse on Cities: History, Current Situation, and Application Perspectives. Appl. Sci., 12.10.3390/app122412820 ↗
25. A proactive role of IoT devices in building smart cities10.1016/j.iotcps.2021.08.001 ↗
26. Effective multiple cancer disease diagnosis frameworks for improved healthcare using machine learning10.1016/j.measurement.2021.109145 ↗
27. Diagnosis of Gastric Cancer Using Machine Learning Techniques in Healthcare Sector: A Survey10.31449/inf.v45i7.3633 ↗
28. Meraghni, S., Benaggoune, K., Al Masry, Z., Terrissa, L.S., Devalland, C., and Zerhouni, N. (2021). Intelligent Computing: Proceedings of the 2021 Computing Conference, Springer International Publishing. [1st ed.].
29. Blocknet: Beyond reliable spatial digital twins to parallel metaverse10.1016/j.patter.2022.100468 ↗
30. Song, M., Shi, Q., Hu, Q., You, Z., and Chen, L. (2020). On the Architecture and Key Technology for Digital Twin Oriented to Equipment Battle Damage Test Assessment. Electronics, 12.10.3390/electronics12010128 ↗
31. Evaluating the effectiveness of learning design with mixed reality (MR) in higher education10.1007/s10055-020-00427-9 ↗
32. From 3D models to 3D prints: An overview of the processing pipeline10.1111/cgf.13147 ↗
33. Fritsch, D., and Klein, M. (November, January 31). 3D and 4D modeling for AR and VR app developments. Proceedings of the 23rd International Conference on Virtual System & Multimedia (VSMM), Dublin, Ireland.
34. Fast weather simulation for inverse procedural design of 3d urban models10.1145/3072959.2999534 ↗
35. Screened poisson surface reconstruction10.1145/2487228.2487237 ↗
36. Chabra, R., Lenssen, J.E., Ilg, E., Schmidt, T., Straub, J., Lovegrove, S., and Newcombe, R. (2020, January 23–28). Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK.10.1007/978-3-030-58526-6_36 ↗
37. Sparsity-based algorithm for condition assessment of rotating machinery using internal encoder data10.1109/TIE.2019.2941172 ↗
38. Li, T., Wen, X., Liu, Y.S., Su, H., and Han, Z. (2022, January 21–24). Learning deep implicit functions for 3D shapes with dynamic code clouds. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA.10.1109/CVPR52688.2022.01250 ↗
39. Guillard, B., Stella, F., and Fua, P. (2022, January 23–27). Meshudf: Fast and differentiable meshing of unsigned distance field networks. Proceedings of the Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel.10.1007/978-3-031-20062-5_33 ↗
40. Atzmon, M., and Lipman, Y. (2020, January 13–19). Sal: Sign agnostic learning of shapes from raw data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.10.1109/CVPR42600.2020.00264 ↗
41. Atzmon, M., and Lipman, Y. (2020). Sald: Sign agnostic learning with derivatives. arXiv.
42. Gropp, A., Yariv, L., Haim, N., Atzmon, M., and Lipman, Y. (2020). Implicit geometric regularization for learning shapes. arXiv.
43. Ma, B., Han, Z., Liu, Y.S., and Zwicker, M. (2020). Neural-pull: Learning signed distance functions from point clouds by learning to pull space onto surfaces. arXiv.
44. Tang, Y.M., and Ho, H.L. (2020). Mixed Reality and Three-Dimensional Computer Graphics, IntechOpen.
45. Van Holland, L., Stotko, P., Krumpen, S., Klein, R., and Weinmann, M. (2020). Efficient 3D Reconstruction, Streaming and Visualization of Static and Dynamic Scene Parts for Multi-client Live-telepresence in Large-scale Environments. arXiv.
46. Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., and Li, H. (November, January 27). Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea.
47. Yang, X., Lin, G., Chen, Z., and Zhou, L. (2023, January 18–22). Neural Vector Fields: Implicit Representation by Explicit Learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.10.1109/CVPR52729.2023.01605 ↗
48. Hong, F., Chen, Z., Lan, Y., Pan, L., and Liu, Z. (2022). Eva3d: Compositional 3d human generation from 2d image collections. arXiv.
49. Geometry-aware Two-scale PIFu Representation for Human Reconstruction
50. A walk in the black-box: 3D visualization of large neural networks in virtual reality10.1007/s00521-022-07608-4 ↗
51. Facilitating learning in immersive virtual reality: Segmentation, summarizing, both or none?10.1111/jcal.12741 ↗
52. Deep learning-based object detection in augmented reality: A systematic review10.1016/j.compind.2022.103661 ↗
53. A digital-twin-assisted fault diagnosis using deep transfer learning10.1109/ACCESS.2018.2890566 ↗
54. Wei, Y., Wei, Z., Rao, Y., Li, J., Zhou, J., and Lu, J. (2022, January 23–28). Lidar distillation: Bridging the beam-induced domain gap for 3d object detection. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-031-19842-7_11 ↗
55. Deep generative models in engineering design: A review10.1115/1.4053859 ↗