journal article·2024-12-16Open access

Three-Dimensional Shape Reconstruction from Digital Freehand Design Sketching Based on Deep Learning Techniques

Ding Zhou, Guohua Wei, Xiaojun Yuan

Applied Sciences· Volume 14· Issue 24

Abstract

This paper proposes a method for 3D reconstruction from Freehand Design Sketching (FDS) in architecture and industrial design. The implementation begins by extracting features from the FDS using the self-supervised learning model DINO, followed by the continuous Signed Distance Function (SDF) regression as an implicit representation through a Multi-Layer Perceptron network. Taking eyeglass frames as an example, the 2D contour and freehand sketch optimize the alignment by their geometrical similarity while exploiting symmetry to improve reconstruction accuracy. Experiments demonstrate that this method can effectively reconstruct high-quality 3D models of eyeglass frames from 2D freehand sketches, outperforming existing deep learning-based 3D reconstruction methods. This research offers practical information for understanding 3D modeling methodology for FDS, triggering multiple modes of design creativity and efficient scheme adjustments in industrial or architectural conceptual design. In conclusion, this novel approach integrates self-supervised learning and geometric optimization to achieve unprecedented fidelity in 3D reconstruction from FDS, setting a new benchmark for AI-driven design processes in industrial and architectural applications.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: 3D Shape Modeling and Analysis · Advanced Numerical Analysis Techniques · 3D Surveying and Cultural Heritage

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

School of System Design and Intelligent Manufacturing, Southern University of Science and Technology, Shenzhen 518051, China

National Key Laboratory of Wireless Communications, University of Electronic Science and Technology of China, Chengdu 611731, China

Funding

2022 Stable Support Plan Program for Shenzhen-based Universities

References (54)

Show source-provided references
1. Wang, Y., Wang, Q., and Xu, F. (2020, January 23–28). Eyeglasses 3D Shape Reconstruction from a Single Face Image. Proceedings of the 16th European Conference on Computer Vision (ECCV 2020), Glasgow, UK. Part XXV 16.10.1007/978-3-030-58595-2_23 ↗
2. Automatic 3D cad models reconstruction from 2D orthographic drawings10.1016/j.cag.2023.05.021 ↗
3. Lifting freehand concept sketches into 3D10.1145/3414685.3417851 ↗
4. Opensketch: A richly-annotated dataset of product design sketches10.1145/3355089.3356533 ↗
5. Bae, S.H., Balakrishnan, R., and Singh, K. (2009, January 4–7). Everybody loves sketch: 3D sketching for a broader audience. Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, Victoria, BC, Canada.10.1145/1622176.1622189 ↗
6. (2024, November 29). Advantages And Disadvantages of Using Computer Aided Design (CAD). Available online: https://www.arcvertex.com/article/advantages-and-disadvantages-of-using-computer-aided-design-cad/.
7. 3D reconstruction of non-textured surface by combining shape from shading and stereovision10.1016/j.measurement.2021.110029 ↗
8. Inferring mirror symmetric 3D shapes from sketches10.1016/j.cad.2012.10.013 ↗
9. Using vanishing points for camera calibration and coarse 3D reconstruction from a single image10.1007/PL00013394 ↗
10. Lun, Z., Gadelha, M., Kalogerakis, E., Maji, S., and Wang, R. (2017, January 25–28). 3D shape reconstruction from sketches via multi-view convolutional networks. Proceedings of the 2017 International Conference on 3D Vision (3DV), Pasadena, CA, USA.10.1109/3DV.2017.00018 ↗
11. Learning to simplify: Fully convolutional networks for rough sketch cleanup
12. Mastering sketching: Adversarial augmentation for structured prediction
13. Strokeaggregator: Consolidating raw sketches into artist-intended curve drawings10.1145/3197517.3201314 ↗
14. Deep learning-based 3D reconstruction: A survey10.1007/s10462-023-10399-2 ↗
15. From 3D reconstruction to virtual reality: A complete methodology for digital archaeological exhibition10.1016/j.culher.2009.02.006 ↗
16. A review of digital industrial and product design methods in UK higher education
17. Architecture as drawing
18. Analysing architecture through drawing10.1080/09613210600879881 ↗
19. A taxonomic classification of visual design representations used by industrial designers and engineering designers
20. Sketching reality: Realistic interpretation of architectural designs10.1145/1356682.1356684 ↗
21. Richards, J. (2014). Freehand renaissance: Concept sketching for a digital age. Res. Rec., 11–17.
22. Richards, J. (2013). Freehand Drawing and Discovery, Enhanced Edition: Urban Sketching and Concept Drawing for Designers, John Wiley & Sons. [1st ed.].
23. Eissen, K., and Steur, R. (2011). Sketching: The Basics, Bis Publishers.
24. An Optimization-Based Approach to the Interpretation of Single Line Drawings as 3D Wire Frames10.1007/BF00129683 ↗
25. Optimization-based reconstruction of a 3D object from a single freehand line drawing10.1016/0010-4485(95)00081-X ↗
26. Sketching folds: Developable surfaces from non-planar silhouettes10.1145/2749458 ↗
27. Implicit meshes for surface reconstruction
28. Recovering three-dimensional shape from a single image of curved objects10.1109/34.24791 ↗
29. GeneCGAN: A conditional generative adversarial network based on genetic tree for point cloud reconstruction10.1016/j.neucom.2021.07.087 ↗
30. Zang, Y., Ding, C., Chen, T., Mao, P., and Hu, W. (2023). Deep3DSketch++: High-Fidelity 3D Modeling from Single Free-hand Sketches. arXiv.10.1109/SMC53992.2023.10393936 ↗
31. Li, C., Pan, H., Liu, Y., Tong, X., Sheffer, A., and Wang, W. (2018, January 12–15). Robust flow-guided neural prediction for sketch-based freeform surface modeling. Proceedings of the ACM SIGGRAPH Asia, Brisbane, Australia.10.1145/3272127.3275051 ↗
32. Wang, J., Lin, J., Yu, Q., Liu, R., Chen, Y., and Yu, S.X. (2020). 3D shape reconstruction from free-hand sketches. arXiv.
33. Zhang, S.H., Guo, Y.C., and Gu, Q.W. (2021, January 19–25). Sketch2Model: View-aware 3D Modeling from Single Free-hand Sketch. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.10.1109/CVPR46437.2021.00595 ↗
34. Towards Pract. Sketch-Based 3D Shape Gener. Role Prof. Sketches10.1109/TCSVT.2020.3040900 ↗
35. Delanoy, J., Aubry, M., Isola, P., Efros, A.A., and Bousseau, A. 3D sketching using multi-view deep volumetric prediction. Proceedings of the ACM on Computer Graphics and Interactive Techniques.
36. Reconstr. 3D Shapes Mult. Sketches Using Direct Shape Optimization10.1109/TIP.2020.3018865 ↗
37. Guillard, B., Remelli, E., Yvernay, P., and Fua, P. (2021, January 10–17). Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada.10.1109/ICCV48922.2021.01278 ↗
38. Remelli, E., Lukoianov, A., Richter, S., Guillard, B., Bagautdinov, T., Baque, P., and Fua, P. (2020, January 6–12). MeshSDF: Differentiable Iso-Surface Extraction. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada.
39. Lorensen, W., and Cline, H. (1987, January 27–31). Marching Cubes: A High Resolution 3D Surface Construction Algorithm. Proceedings of the ACM SIGGRAPH, Anaheim, CA, USA.10.1145/37401.37422 ↗
40. A Near optimal Isosurface Extraction Algorithm Using the Span Space10.1109/2945.489388 ↗
41. Fast isosurface generation using the volume thinning algorithm10.1109/2945.910819 ↗
42. Topological considerations in isosurface generation10.1145/195826.195828 ↗
43. A survey of the marching cubes algorithm10.1016/j.cag.2006.07.021 ↗
44. Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., and El-Nouby, A. (2023). Dinov2: Learning robust visual features without supervision. arXiv.
45. Park, J.J., Florence, P., Straub, J., Newcombe, R.A., and Lovegrove, S. (2019, January 16–20). DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA.10.1109/CVPR.2019.00025 ↗
46. Ravi, N., Reizenstein, J., Novotny, D., Gordon, T., Lo, W.Y., Johnson, J., and Gkioxari, G. (2020). Accelerating 3D Deep Learning with PyTorch3D. arXiv.
47. (2024, July 25). Pytorch Lightning. Available online: https://github.com/Lightning-AI/pytorch-lightning.
48. Ioffe, S., and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. arXiv.
49. Salimans, T., and Kingma, D.P. (2016, January 5–10). Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks. Proceedings of the Advances in Neural Information Processing Systems, Barcelona, Spain.
50. Kingma, D.P., and Ba, J. (2014). Adam: A Method for Stochastic Optimization. arXiv.
51. Saito, T., and Takahashi, T. (1990, January 4–8). Comprehensible Rendering of 3-D Shapes. Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, Dallas, TX, USA.10.1145/97879.97901 ↗
52. (2024, December 04). Incorporating Asymmetry into Your Architectural Design—Architecture Plus LLC. Available online: https://care-can.squarespace.com/blog/2021/10/2/incorporating-asymmetry-into-your-architectural-design.
53. Vectorization of line drawings via poly vector fields10.1145/3202661 ↗
54. Novel biological therapies targeting heart failure: Myocardial rejuvenation10.1016/j.hfc.2016.03.008 ↗