Symbol Detection in Mechanical Engineering Sketches: Experimental Study on Principle Sketches with Synthetic Data Generation and Deep Learning
Source abstract
Abstract
Digital transformation is omnipresent in our daily lives and its impact is noticeable through new technologies, like smart devices, AI-Chatbots or the changing work environment. This digitalization also takes place in product development, with the integration of many technologies, such as Industry 4.0, digital twins or data-driven methods, to improve the quality of new products and to save time and costs during the development process. Therefore, the use of data-driven methods reusing existing data has great potential. However, data from product design are very diverse and strongly depend on the respective development phase. One of the first few product representations are sketches and drawings, which represent the product in a simplified and condensed way. But, to reuse the data, the existing sketches must be found with an automated approach, allowing the contained information to be utilized. One approach to solve this problem is presented in this paper, with the detection of principle sketches in the early phase of the development process. The aim is to recognize the symbols in these sketches automatically with object detection models. Therefore, existing approaches were analyzed and a new procedure developed, which uses synthetic training data generation. In the next step, a total of six different data generation types were analyzed and tested using six different one- and two-stage detection models. The entire procedure was then evaluated on two unknown test datasets, one focusing on different gearbox variants and a second dataset derived from CAD assemblies. In the last sections the findings are discussed and a procedure with high detection accuracy is determined.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Industrial Vision Systems and Defect Detection · Manufacturing Process and Optimization · Additive Manufacturing Materials and Processes
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Engineering Design, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058 Erlangen, Germany
References (100+)
Show source-provided references
1. Data-driven engineering design: A systematic review using scientometric approach10.1016/j.aei.2022.101774 ↗
2. Achieving Benefits with Design Reuse in Manufacturing Industry10.1016/j.procir.2016.04.173 ↗
3. Isaksson, O., Hallstedt, S.I., and Rönnbäck, A.Ö. (2018, January 14–17). Digitalisation, sustainability and servitisation: Consequences on product development capabilities in manufacturing firms. Proceedings of the DS 91: NordDesign 2018, Linköping, Sweden.
4. Hahne, M. (2019). Systematisches Konstruieren: Praxisnah und Prägnant, Springer.10.1007/978-3-658-25905-1 ↗
5. Produktentwicklung und Projektmanagement (1997). Konstruktionsmethodik—Methodisches Entwickeln von Lösungsprinzipien, VDI.
6. Roth, K. (2000). Konstruieren mit Konstruktionskatalogen: Band 1: Konstruktionslehre, Springer. [3rd ed.]. Erweitert und Neu Gestaltet.10.1007/978-3-642-17466-7_1 ↗
7. New trends on digitisation of complex engineering drawings10.1007/s00521-018-3583-1 ↗
8. Valveny, E., and Dosch, P. (2003, January 30–31). Symbol recognition contest: A synthesis. Proceedings of the International Workshop on Graphics Recognition, Barcelona, Spain.
9. Goyal, S., Mistry, V., Chattopadhyay, C., and Bhatnagar, G. (2019, January 20–25). BRIDGE: Building Plan Repository for Image Description Generation, and Evaluation. Proceedings of the 2019 International Conference on Document Analysis and Recognition (ICDAR), Sydney, Australia.10.1109/ICDAR.2019.00174 ↗
10. Generation of synthetic documents for performance evaluation of symbol recognition & spotting systems10.1007/s10032-010-0120-x ↗
11. Redmon, J., Divvala, S., Girshick, R., and Farhadi, A. (July, January 26). You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA.
12. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich feature hierarchies for accurate object detection and semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
13. Pereira, F., Burges, C., Bottou, L., and Weinberger, K. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, Curran Associates, Inc.
14. Girshick, R. (2015, January 7–13). Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile.10.1109/ICCV.2015.169 ↗
15. Ren, S., He, K., Girshick, R., and Sun, J. (2015, January 7–12). Faster r-cnn: Towards real-time object detection with region proposal networks. Proceedings of the Advances in Neural Information Processing Systems 28 (NIPS 2015), Montreal, QC, Canada.
16. He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017, January 22–29). Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.322 ↗
17. Rezvanifar, A., Cote, M., and Albu, A.B. (2020, January 14–19). Symbol Spotting on Digital Architectural Floor Plans Using a Deep Learning-based Framework. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA.10.1109/CVPRW50498.2020.00292 ↗
18. Symbol spotting for architectural drawings: State-of-the-art and new industry-driven developments10.1186/s41074-019-0055-1 ↗
19. Stefenon, S.F., Cristoforetti, M., and Cimatti, A. (2023, January 6–9). Towards Automatic Digitalization of Railway Engineering Schematics. Proceedings of the International Conference of the Italian Association for Artificial Intelligence, Rome, Italy.10.1007/978-3-031-47546-7_31 ↗
20. Huang, W., Sun, Q., Yu, A., Guo, W., Xu, Q., Wen, B., and Xu, L. (2023). Leveraging Deep Convolutional Neural Network for Point Symbol Recognition in Scanned Topographic Maps. ISPRS Int. J.-Geo-Inf., 12.10.3390/ijgi12030128 ↗
21. Surikov, I.Y., Nakhatovich, M.A., Belyaev, S.Y., and Savchuk, D.A. (2020, January 26–27). Floor Plan Recognition and Vectorization Using Combination UNet, Faster-RCNN, Statistical Component Analysis and Ramer-Douglas-Peucker. Proceedings of the International Conference on Computing Science, Communication and Security, Gujarat, India.10.1007/978-981-15-6648-6_2 ↗
22. Sarkar, S., Pandey, P., and Kar, S. (2022). Automatic Detection and Classification of Symbols in Engineering Drawings. arXiv.
23. Mishra, S., Hashmi, K.A., Pagani, A., Liwicki, M., Stricker, D., and Afzal, M.Z. (2021). Towards Robust Object Detection in Floor Plan Images: A Data Augmentation Approach. Appl. Sci., 11.10.20944/preprints202110.0089.v1 ↗
24. Shehzadi, T., Hashmi, K.A., Pagani, A., Liwicki, M., Stricker, D., and Afzal, M.Z. (2022). Mask-Aware Semi-Supervised Object Detection in Floor Plans. Appl. Sci., 12.10.20944/preprints202209.0025.v1 ↗
25. Goyal, S. (2021). Fine Grained Feature Representation Using Computer Vision Techniques for Understanding Indoor Space. [Ph.D. Thesis, Indian Institute of Technology Jodhpur and Jodhpur and Computer Science and Engineering].
26. Elyan, E., Jamieson, L., and Ali-Gombe, A. (2020). Deep Learning for Symbols Detection and Classification in Engineering Drawings, Elsevier.10.1016/j.neunet.2020.05.025 ↗
27. Dzhusupova, R., Banotra, R., Bosch, J., and Olsson, H.H. (2022, January 6–9). Pattern Recognition Method for Detecting Engineering Errors on Technical Drawings. Proceedings of the 2022 IEEE World AI IoT Congress (AIIoT), Seattle, WA, USA.10.1109/AIIoT54504.2022.9817294 ↗
28. Xiao, X., Li, Z., Zhao, S., Yang, L., Zhao, F., and Ge, C. (2023, January 1–4). Improved P&ID Symbol Detection Algorithm Based on YOLOv5 Network. Proceedings of the 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Honolulu, HI, USA.10.1109/SMC53992.2023.10394450 ↗
29. Gajbhiye, S.M., Bhamre, S., Tadepalli, L.T., Pillai, M., and Uplaonkar, D. (2023, January 24–25). Advancing P&ID Digitization with YOLOv5. Proceedings of the 2023 International Conference on Integrated Intelligence and Communication Systems (ICIICS), Kalaburagi, India.10.1109/ICIICS59993.2023.10421368 ↗
30. Ismail, M.H.A.B. (2021, January 16). Identification of Objects in Oilfield Infrastructure using Engineering Diagram and Machine Learning Methods. Proceedings of the 2021 IEEE Symposium on Computers & Informatics (ISCI), 2021, Kuala Lumpur, Malaysia.10.1109/ISCI51925.2021.9633745 ↗
31. Liu, S., Li, Z., Zhao, S., Yang, L., Zhao, F., and Ge, C. (2023, January 17–21). A Complete Piping Identification Solution for Piping and Instrumentation Diagrams. Proceedings of the 2023 IEEE International Conference on High Performance Computing & Communications, Data Science & Systems, Smart City & Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), Melbourne, Australia.10.1109/HPCC-DSS-SmartCity-DependSys60770.2023.00011 ↗
32. Yun, D.Y., Seo, S.K., Zahid, U., and Lee, C.J. (2020). Deep Neural Network for Automatic Image Recognition of Engineering Diagrams. Appl. Sci., 10.10.3390/app10114005 ↗
33. Zhang, Y., Cai, J., and Cai, H. (2020). CNN-Based Symbol Recognition in Piping Drawings, American Society of Civil Engineers.10.1061/9780784482865.061 ↗
34. Kang, D.H., Choi, S.W., Lee, E.B., and Kang, S.O. (2024). Auto-Routing Systems (ARSs) with 3D Piping for Sustainable Plant Projects Based on Artificial Intelligence (AI) and Digitalization of 2D Drawings and Specifications. Sustainability, 16.10.3390/su16072770 ↗
35. Graph-Based Object Detection Enhancement for Symbolic Engineering Drawings10.1007/978-3-030-86198-8_6 ↗
36. Song, A., Kun, H., Peng, B., Chen, R., Zhao, K., Qiu, J., and Wang, K. (2021, January 22–24). EDRS: An Automatic System to Recognize Electrical Drawings. Proceedings of the 2021 China Automation Congress (CAC), Beijing, China.10.1109/CAC53003.2021.9728054 ↗
37. Haar, C., Kim, H., and Koberg, L. (2022, January 19–23). AI-Based Engineering and Production Drawing Information Extraction. Proceedings of the International Conference on Flexible Automation and Intelligent Manufacturing, Detroit, MI, USA.10.1007/978-3-031-18326-3_36 ↗
38. Lin, Y.H., Ting, Y.H., Huang, Y.C., Cheng, K.L., and Jong, W.R. (2023). Integration of Deep Learning for Automatic Recognition of 2D Engineering Drawings. Machines, 11.10.3390/machines11080802 ↗
39. Kashevnik, A., Ali, A., and Mayatin, A. (2023, January 27–31). AI-Based Method for Frame Detection in Engineering Drawings. Proceedings of the 2023 International Russian Smart Industry Conference (SmartIndustryCon), Sochi, Russia.10.1109/SmartIndustryCon57312.2023.10110731 ↗
40. Brock, A., Lim, T., Ritchie, J.M., and Weston, N. (2017, January 6–9). ConvNet-Based Optical Recognition for Engineering Drawings. Proceedings of the ASME 2017 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Cleveland, OH, USA.10.1115/DETC2017-68186 ↗
41. From Sketches to Graphs: A Deep Learning Based Method for Detection and Contextualisation of Principle Sketches in the Early Phase of Product Development10.1017/pds.2023.198 ↗
42. Detection and classification of symbols in principle sketches using deep learning10.1017/pds.2021.118 ↗
43. Seff, A., Ovadia, Y., Zhou, W., and Adams, R.P. (2020). Sketchgraphs: A large-scale dataset for modeling relational geometry in computer-aided design. arXiv.
44. Onshape, P. (2024, July 08). API-integration with Onshape. Available online: https://www.onshape.com/de/features/integrations.
45. Clark, A. (2024, April 23). Pillow (PIL Fork) Documentation. Available online: https://buildmedia.readthedocs.org/media/pdf/pillow/latest/pillow.pdf.
46. (2024, April 23). github. object-detection. Available online: https://github.com/topics/object-detection.
47. Roth, K. (2013). Konstruieren mit Konstruktionskatalogen: Band 2: Kataloge, Springer.
48. Roth, K. (1996). Konstruieren mit Konstruktionskatalogen: Band 3: Verbindungen und Verschlüsse, Lösungsfindung, Springer. [2nd ed.]. Wesentlich Erweitert und neu Gestaltet; Springer eBook Collection Computer Science and Engineering.
49. Labisch, S., and Weber, C. (2005). Technisches Zeichnen: Intensiv und Effektiv Lernen und üben, überarb. aufl; Studium, Vieweg. [2nd ed.].
50. (1995). Vereinfachte Darstellungen in der Kinematik (Standard No. DIN ISO 3952-2:1995 DE).
51. List, R. (2009). CATIA V5—Grundkurs für Maschinenbauer: Bauteil- und Baugruppenkonstruktion, Zeichnungsableitung, Studium, Vieweg + Teubner. [4th ed.]. aktualisierte und erw. aufl.10.1007/978-3-8348-9678-0_10 ↗
52. Madsen, D.A., Madsen, D.P., Standiford, K., and Krulikowski, A. (2017). Engineering Drawing & Design, Cengage Learning. [6th ed.].
53. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollar, P., and Zitnick, C.L. (2014, January 6–12). Microsoft coco: Common objects in context. Proceedings of the European Conference on Computer Vision, Zurich, Switzerland.10.1007/978-3-319-10602-1_48 ↗
54. Abdulla, W. (2024, April 23). Mask R-CNN for Object Detection and Instance Segmentation on Keras and TensorFlow. Available online: https://github.com/matterport/Mask_RCNN.
55. Jocher, G., Stoken, A., Chaurasia, A., Borovec, J., Code, N., Xie, T., Kwon, Y., Michael, K., Changyu, L., and Fang, J. (2021). ultralytics/yolov5: v6.0—YOLOv5n ’Nano’ models, Roboflow integration, TensorFlow export, OpenCV DNN support. Zenodo.
56. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., and Girshick, R. (2024, April 23). Detectron2. Available online: https://github.com/facebookresearch/detectron2.
57. Content-based CAD assembly model retrieval: Survey and future challenges10.1016/j.cad.2019.03.005 ↗
58. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014, January 8–13). Generative adversarial nets. Proceedings of the Advances in Neural Information Processing Systems, Montreal, QC, Canada.
59. Vahdat, A., and Kautz, J. (2020, January 6–12). NVAE: A deep hierarchical variational autoencoder. Proceedings of the Advances in Neural Information Processing Systems 2020, virtual.
60. Kingma, D.P., and Welling, M. (2014). Auto-encoding variational bayes. arXiv.
61. Object detection in 20 years: A survey10.1109/JPROC.2023.3238524 ↗
62. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., and Berg, A.C. (2016, January 11–14). Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands.10.1007/978-3-319-46448-0_2 ↗
63. Lin, T.Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017, January 22–29). Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy.10.1109/ICCV.2017.324 ↗
64. Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. (2017, January 21–26). Feature pyramid networks for object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA.10.1109/CVPR.2017.106 ↗
65. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. (2020, January 23–28). End-to-end object detection with transformers. Proceedings of the European Conference on Computer Vision, Glasgow, UK.10.1007/978-3-030-58452-8_13 ↗
66. Law, H., and Deng, J. (2018, January 8–14). Cornernet: Detecting objects as paired keypoints. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.10.1007/978-3-030-01264-9_45 ↗
67. A Threshold Selection Method from Gray-Level Histograms10.1109/TSMC.1979.4310076 ↗
68. A system for interpretation of line drawings10.1109/34.58870 ↗
69. Tanigawa, S., Hori, O., and Shimotsuji, S. (1994, January 9–13). Precise line detection from an engineering drawing using a figure fitting method based on contours and skeletons. Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5), Jerusalem, Israel.
70. Skeleton generation of engineering drawings via contour matching10.1016/0031-3203(94)90058-2 ↗
71. den Hartog, J.E., and ten Kate, T.K. (1994, January 9–13). Finding arrows in utility maps using a neural network. Proceedings of the 12th IAPR International Conference on Pattern Recognition (Cat. No.94CH3440-5), Jerusalem, Israel.
72. Messmer, B.T., and Bunke, H. (1996). Automatic learning and recognition of graphical symbols in engineering drawings. Graphics Recognition Methods and Applications, Springer.10.1007/3-540-61226-2_11 ↗
73. A new algorithm for line image vectorization10.1016/S0031-3203(97)00157-X ↗
74. Adaptive document image binarization10.1016/S0031-3203(99)00055-2 ↗
75. Robust and accurate vectorization of line drawings10.1109/TPAMI.2006.127 ↗
76. Multi-Level Block Information Extraction in Engineering Drawings Based on Depth-First Algorithm
77. Rezaei, S.B., Shanbehzadeh, J., and Sarrafzadeh, A. (2017, January 15–17). Adaptive document image skew estimation. Proceedings of the International MultiConference of Engineers and Computer Scientists 2017—IMECS 2017, Hong Kong, China.
78. Liu, T., Hua, Q., Yuan, S., Yin, L., and Cheng, G. (2019, January 21–22). Anchor Point based Hough Transformation for Automatic Line Detection of Engineering Drawings. Proceedings of the 2019 WRC Symposium on Advanced Robotics and Automation (WRC SARA), Beijing, China.10.1109/WRC-SARA.2019.8931934 ↗
79. Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.M. (2020). Deep Vectorization of Technical Drawings, Springer International Publishing.
80. Yang, D.S., Webster, J.L., Renmdell, L.A., Garrett, J.H., and Shaw, D.S. (1993, January 8–11). Management of graphical symbols in a CAD environment: A neural network approach. Proceedings of the 1993 IEEE Conference on Tools with Al (TAI-93), Boston, MA, USA.
81. Symbol recognition in a CAD environment using a neural network10.1142/S0218213094000091 ↗
82. A system to understand hand-drawn floor plans using subgraph isomorphism and Hough transform10.1007/s001380050068 ↗
83. Ah-Soon, C. (1998). A constraint network for symbol detection in architectural drawings. Graphics Recognition Algorithms and Systems, Springer.10.1007/3-540-64381-8_41 ↗
84. A complete system for the analysis of architectural drawings10.1007/PL00010901 ↗
85. Architectural symbol recognition using a network of constraints10.1016/S0167-8655(00)00091-X ↗
86. Symbol recognition by error-tolerant subgraph matching between region adjacency graphs10.1109/34.954603 ↗
87. An object-oriented progressive-simplification-based vectorization system for engineering drawings: Model, algorithm, and performance10.1109/TPAMI.2002.1023802 ↗
88. A model for image generation and symbol recognition through the deformation of lineal shapes10.1016/S0167-8655(03)00144-2 ↗
89. Using engineering drawing interpretation for automatic detection of version information in CADD engineering drawing10.1016/j.autcon.2004.08.004 ↗
90. Automatic analysis and integration of architectural drawings10.1007/s10032-006-0029-6 ↗
91. Lu, T., Yang, Y., Yang, R., and Cai, S. (2008, January 18–20). Knowledge Extraction from Structured Engineering Drawings. Proceedings of the 2008 Fifth International Conference on Fuzzy Systems and Knowledge Discovery, Jinan, China.10.1109/FSKD.2008.184 ↗
92. Le Bodic, P., Locteau, H., Adam, S., Héroux, P., Lecourtier, Y., and Knippel, A. (2009, January 26–29). Symbol Detection Using Region Adjacency Graphs and Integer Linear Programming. Proceedings of the 2009 10th International Conference on Document Analysis and Recognition, Barcelona, Spain.10.1109/ICDAR.2009.202 ↗
93. Symbol spotting in vectorized technical drawings through a lookup table of region strings10.1007/s10044-009-0161-2 ↗
94. Nayef, N., and Breuel, T.M. (2011, January 18–21). Statistical Grouping for Segmenting Symbols Parts from Line Drawings, with Application to Symbol Spotting. Proceedings of the 2011 International Conference on Document Analysis and Recognition, Beijing, China.10.1109/ICDAR.2011.81 ↗
95. Barducci, A., and Marinai, S. (2012, January 11–15). Object recognition in floor plans by graphs of white connected components. Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan.
96. Nayef, N. (2013). Geometric-based Symbol Spotting and Retrieval in Technical Line Drawings. [Ph.D. Thesis, Technische Universität Kaiserslautern].
97. Nayef, N., and Breuel, T.M. (2013). Building a Symbol Library from Technical Drawings by Identifying Repeating Patterns. Graphics Recognition. New Trends and Challenges, Springer.10.1007/978-3-642-36824-0_7 ↗
98. Nayef, N., and Breuel, T.M. (2013, January 3–7). Combining geometric matching with SVM to improve symbol spotting. Proceedings of the IS & T/SPIE Electronic Imaging, 2013, Burlingame, CA, USA.10.1117/12.2002795 ↗
99. Nayef, N., and Breuel, T.M. (2013, January 5–7). Efficient symbol retrieval by building a symbol index from a collection of line drawings. Proceedings of the IS&T-SPIE Electronic Imaging Symposium, Burlingame, CA, USA.10.1117/12.2008532 ↗
100. A symbol spotting approach in graphical documents by hashing serialized graphs10.1016/j.patcog.2012.10.003 ↗