An Intelligent Manufacturing Approach Based on a Novel Deep Learning Method for Automatic Machine and Working Status Recognition
Source abstract
Abstract
Smart manufacturing uses robots and artificial intelligence techniques to minimize human interventions in manufacturing activities. Inspection of the machine’ working status is critical in manufacturing processes, ensuring that machines work correctly without any collisions and interruptions, e.g., in lights-out manufacturing. However, the current method heavily relies on workers onsite or remotely through the Internet. The existing approaches also include a hard-wired robot working with a computer numerical control (CNC) machine, and the instructions are followed through a pre-program path. Currently, there is no autonomous machine tending application that can detect and act upon the operational status of a CNC machine. This study proposes a deep learning-based method for the CNC machine detection and working status recognition through an independent robot system without human intervention. It is noted that there is often more than one machine working in a representative industrial environment. Therefore, the SiameseRPN method is developed to recognize and locate a specific machine from a group of machines. A deep learning-based text recognition method is designed to identify the working status from the human–machine interface (HMI) display.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Industrial Vision Systems and Defect Detection · Robot Manipulation and Learning · Manufacturing Process and Optimization
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
Laboratory of Intelligent Manufacturing, Design and Automation (LIMDA), Department of Mechanical Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada
Department of Mechanical and Energy Engineering, Southern University of Science and Technology, Shenzhen 518055, China
Funding
NSERC
References (49)
Show source-provided references
1. Intelligent vision-based online inspection system of screw-fastening operations in light-gauge steel frame manufacturing10.1007/s00170-020-05695-y ↗
2. Current Researches and Future Development Trend of Intelligent Robot: A Review10.1007/s11633-018-1115-1 ↗
3. Learning Stable Nonlinear Dynamical Systems with Gaussian Mixture Models10.1109/TRO.2011.2159412 ↗
4. Methods and Technologies for the Implementation of Large-Scale Robot Tactile Sensors10.1109/TRO.2011.2132930 ↗
5. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection10.1177/0278364917710318 ↗
6. Finn, C., and Levine, S. (June, January 29). Deep visual foresight for planning robot motion. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989324 ↗
7. Vision-Based Associative Robotic Recognition of Working Status in Autonomous Manufacturing Environment10.1016/j.procir.2021.11.259 ↗
8. Huang, Y., Wu, H., Liu, H., and Yin, Z. (2017). An NC Code Based Machining Movement Simulation Method for a Parallel Robotic Machine, Springer International Publishing.
9. Real-time visual detection and correction of automatic screw operations in dimpled light-gauge steel framing with pre-drilled pilot holes10.1016/j.promfg.2019.06.204 ↗
10. Matheson, E., Minto, R., Zampieri, E.G.G., Faccio, M., and Rosati, G. (2019). Human–Robot Collaboration in Manufacturing Applications: A Review. Robotics, 8.10.3390/robotics8040100 ↗
11. Jia, F., Tzintzun, J., and Ahmad, R. (2020). An Improved Robot Path Planning Algorithm for a Novel Self-Adapting Intelligent Machine Tending Robotic System, Springer International Publishing.10.1007/978-3-030-45402-9_7 ↗
12. Intelligent knowledge-based system to improve injection molding process
13. Agglomeration and driving factors of regional innovation space based on intelligent manufacturing and green economy10.1016/j.eti.2021.101398 ↗
14. Vision-Based Intelligent Perceiving and Planning System of a 7-DoF Collaborative Robot10.1155/2021/5810371 ↗
15. Towards Collaborative Robotics in Top View Surveillance: A Framework for Multiple Object Tracking by Detection Using Deep Learning10.1109/JAS.2020.1003453 ↗
16. Learning-based object detection and localization for a mobile robot manipulator in SME production10.1016/j.rcim.2021.102229 ↗
17. Dalal, N., and Triggs, B. (2005, January 20–25). Histograms of Oriented Gradients for Human Detection. Proceedings of the Computer Vision and Pattern Recognition, San Diego, CA, USA.
18. Felzenszwalb, P., McAllester, D., and Ramanan, D. (2008, January 23–28). A discriminatively trained, multiscale, deformable part model. Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, AK, USA.10.1109/CVPR.2008.4587597 ↗
19. Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014, January 23–28). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA.10.1109/CVPR.2014.81 ↗
20. Girshick, R. (2015, January 7–13). Fast R-CNN. Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile.10.1109/ICCV.2015.169 ↗
21. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks10.1109/TPAMI.2016.2577031 ↗
22. Leal-Taixé, L., Canton-Ferrer, C., and Schindler, K. (July, January 26). Learning by Tracking: Siamese CNN for Robust Target Association. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops, Las Vegas, NV, USA.10.1109/CVPRW.2016.59 ↗
23. Review of Scene Text Detection and Recognition10.1007/s11831-019-09315-1 ↗
24. Scene text detection and recognition10.1007/s11704-015-4488-0 ↗
25. Brisinello, M., Grbic, R., Vranjes, M., and Vranjes, D. (2019, January 23–25). Review on Text Detection Methods on Scene Images. Proceedings of the 2019 International Symposium ELMAR, Zadar, Croatia.10.1109/ELMAR.2019.8918680 ↗
26. Reading Text in the Wild with Convolutional Neural Networks10.1007/s11263-015-0823-z ↗
27. Zhong, Z., Jin, L., Zhang, S., and Feng, Z. (2016). DeepText: A Unified Framework for Text Proposal Generation and Text Detection in Natural Images. arXiv.10.1109/ICASSP.2017.7952348 ↗
28. He, W., Zhang, X.-Y., Yin, F., and Liu, C.-L. (2017, January 22–29). Deep Direct Regression for Multi-oriented Scene Text Detection. Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy.10.1109/ICCV.2017.87 ↗
29. Accurate Scene Text Recognition Based on Recurrent Neural Network
30. An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition10.1109/TPAMI.2016.2646371 ↗
31. Lee, C.-Y., and Osindero, S. (2016, January 27–30). Recursive Recurrent Nets with Attention Modeling for OCR in the Wild. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA.10.1109/CVPR.2016.245 ↗
32. Bahdanau, D., Cho, K., and Bengio, Y. (2015, January 7–9). Neural machine translation by jointly learning to align and translate. Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA.
33. Liao, M., Shi, B., Bai, X., Wang, X., and Liu, W. (2017, January 4–9). TextBoxes: A fast text detector with a single deep neural network. Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, CA, USA.10.1609/aaai.v31i1.11196 ↗
34. Hradiš, M., Kotera, J., Zemčík, P., and Šroubek, F. (2015, January 7–10). Convolutional Neural Networks for Direct Text Deblurring. Proceedings of the British Machine Vision Conference, Swansea, UK.10.5244/C.29.6 ↗
35. Text detection and localization in scene images: A broad review10.1007/s10462-021-10000-8 ↗
36. Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., and Schmidhuber, J. (2008). Unconstrained online handwriting recognition with recurrent neural networks. Advances in Neural Information Processing System, MIT Press.
37. Rotation-invariant convolutional neural networks for galaxy morphology prediction10.1093/mnras/stv632 ↗
38. A Novel Deep Learning-based Automatic Damage Detection and Localization Method for Remanufacturing/Repair
39. ImageNet Large Scale Visual Recognition Challenge10.1007/s11263-015-0816-y ↗
40. Defining the Characteristics Needed for the Cobots Design Used for the Supply of Cnc Machines
41. Al-Hussaini, S., Thakar, S., Kim, H., Rajendran, P., Shah, B.C., Marvel, J.A., and Gupta, S.K. (2020). Human-Supervised Semi-Autonomous Mobile Manipulators for Safely and Efficiently Executing Machine Tending Tasks. arXiv.
42. Annem, V., Rajendran, P., Thakar, S., and Gupta, S.K. (2019, January 10–14). Towards Remote Teleoperation of a Semi-Autonomous Mobile Manipulator System in Machine Tending Tasks. Proceedings of the ASME 2019 14th International Manufacturing Science and Engineering Conference, Erie, PA, USA.10.1115/MSEC2019-3027 ↗
43. Scene Text Detection and Recognition: The Deep Learning Era10.1007/s11263-020-01369-0 ↗
44. Wojna, Z., Gorban, A.N., Lee, D.-S., Murphy, K., Yu, Q., Li, Y., and Ibarz, J. (2017, January 9–15). Attention-Based Extraction of Structured Information from Street View Imagery. Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan.10.1109/ICDAR.2017.143 ↗
45. Nadeem, M.S., Hussain, S., and Kurugollu, F. (2021). Textual Deblurring using Convolutional Neural Network. TechRxiv.10.36227/techrxiv.16760632 ↗
46. Wu, C., Du, H., Wu, Q., and Zhang, S. (2020). Image Text Deblurring Method Based on Generative Adversarial Network. Electronics, 9.10.3390/electronics9020220 ↗
47. Single-Image Reflection Removal Using Deep Learning: A Systematic Review10.1109/ACCESS.2022.3156273 ↗
48. Li, C., Yang, Y., He, K., Lin, S., and Hopcroft, J.E. (2020, January 13–19). Single Image Reflection Removal Through Cascaded Refinement. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA.10.1109/CVPR42600.2020.00362 ↗
49. Li, Y., Liu, M., Yi, Y., Li, Q., Ren, D., and Zuo, W. (2020). Two-Stage Single Image Reflection Removal with Reflection-Aware Guidance. arXiv.