A Review of Deep Reinforcement Learning Approaches for Smart Manufacturing in Industry 4.0 and 5.0 Framework
Source abstract
Abstract
In this review, the industry’s current issues regarding intelligent manufacture are presented. This work presents the status and the potential for the I4.0 and I5.0’s revolutionary technologies. AI and, in particular, the DRL algorithms, which are a perfect response to the unpredictability and volatility of modern demand, are studied in detail. Through the introduction of RL concepts and the development of those with ANNs towards DRL, the potential and variety of these kinds of algorithms are highlighted. Moreover, because these algorithms are data based, their modification to meet the requirements of industry operations is also included. In addition, this review covers the inclusion of new concepts, such as digital twins, in response to an absent environment model and how it can improve the performance and application of DRL algorithms even more. This work highlights that DRL applicability is demonstrated across all manufacturing industry operations, outperforming conventional methodologies and, most notably, enhancing the manufacturing process’s resilience and adaptability. It is stated that there is still considerable work to be carried out in both academia and industry to fully leverage the promise of these disruptive tools, begin their deployment in industry, and take a step closer to the I5.0 industrial revolution.
Source: crossref · Quality: baseline
Research context
OpenAlex topics: Digital Transformation in Industry · Industrial Vision Systems and Defect Detection · Flexible and Reconfigurable Manufacturing Systems
SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.
Affiliations
School of Engineering (ETSI), Systems and Automation Department, University of Sevilla, 41092 Sevilla, Spain
IT Department, IDENER, 41300 Sevilla, Spain
References (100+)
Show source-provided references
1. A review of the meanings and the implications of the Industry 4.0 concept10.1016/j.promfg.2017.09.032 ↗
2. Industry 4.010.1007/s12599-014-0334-4 ↗
3. Meena, M., Wangtueai, S., Mohammed Sharafuddin, A., and Chaichana, T. (2022). The Precipitative Effects of Pandemic on Open Innovation of SMEs: A Scientometrics and Systematic Review of Industry 4.0 and Industry 5.0. J. Open Innov. Technol. Mark. Complex., 8.10.3390/joitmc8030152 ↗
4. (2022, October 10). Industry 5.0—Publications Office of the EU. Available online: https://op.europa.eu/en/publication-detail/-/publication/468a892a-5097-11eb-b59f-01aa75ed71a1/.
5. Industry 4.0 and Industry 5.0—Inception, conception and perception10.1016/j.jmsy.2021.10.006 ↗
6. Crnjac, Z.M., Mladineo, M., Gjeldum, N., and Celent, L. (2022). From Industry 4.0 towards Industry 5.0: A Review and Analysis of Paradigm Shift for the People, Organization and Technology. Energies, 15.10.3390/en15145221 ↗
7. The World Bank (2022, October 11). Manufacturing, Value Added (% of GDP)—World|Data. Available online: https://data.worldbank.org/indicator/NV.IND.MANF.ZS.
8. The World Bank (2022, October 11). Manufacturing, Value Added (% of GDP)—European Union|Data. Available online: https://data.worldbank.org/indicator/NV.IND.MANF.ZS?locations=EU&name_desc=false.
9. Yin, R. (2016). Concept and Theory of Dynamic Operation of the Manufacturing Process. Theory and Methods of Metallurgical Process Integration, Academic Press.10.1016/B978-0-12-809568-3.00012-7 ↗
10. Monitoring and Control of Manufacturing Processes: A Review10.1016/j.procir.2013.06.127 ↗
11. Machine learning in manufacturing: Advantages, challenges, and applications
12. Deep reinforcement learning in production systems: A systematic literature review10.1080/00207543.2021.1973138 ↗
13. Industry 5.0: A survey on enabling technologies and potential applications
14. Bigan, C. (2022). Trends in Teaching Artificial Intelligence for Industry 5.0. Sustainability and Innovation in Manufacturing Enterprises, Springer.10.1007/978-981-16-7365-8_10 ↗
15. Sutton, R.S., and Barto, A.G. (2020). Finitie Markov Decision Processes. Reinforcement Learning: An Introduction, The MIT Press. [2nd ed.]. Available online: http://incompleteideas.net/book/RLbook2020.pdf.
16. Virvou, M., Alepis, E., Tsihrintzis, G.A., and Jain, L.C. (2020). Machine Learning Paradigms, Springer.10.1007/978-3-030-13743-4 ↗
17. Coursera (2022, November 05). 3 Types of Machine Learning You Should Know. Available online: https://www.coursera.org/articles/types-of-machine-learning.
18. Wiering, M., and Otterlo, M. (2012). Reinforcement learning. Adaptation, learning, and optimization. Reinforcement Learning State-of-the-Art, Springer.10.1007/978-3-642-27645-3 ↗
19. A Markovian Decision Process
20. van Otterlo, M., and Wiering, M. (2012). Reinforcement learning and markov decision processes. Reinforcement Learning, Springer.10.1007/978-3-642-27645-3_1 ↗
21. Reward, motivation, and reinforcement learning10.1016/S0896-6273(02)00963-7 ↗
22. A practical guide to multi-objective reinforcement learning and planning10.1007/s10458-022-09552-y ↗
23. Reinforcement learning: Exploration-exploitation dilemma in multi-agent foraging task10.1007/s12597-012-0077-2 ↗
24. Coggan, M. (2004). Exploration and exploitation in reinforcement learning. CRA-W DMP Project, McGill University. Working Paper of the Research Supervised by Prof. Doina Precup.
25. A Survey of Exploration Strategies in Reinforcement Learning
26. Induction and exploitation of subgoal automata for reinforcement learning10.1613/jair.1.12372 ↗
27. Deep Reinforcement Learning for Autonomous Driving: A Survey10.1109/TITS.2021.3054625 ↗
28. Polvara, R., Patacchiola, M., Sharma, S., Wan, J., Manning, A., Sutton, R., and Cangelosi, A. (2017). Autonomous quadrotor landing using deep reinforcement learning. ArXiv.
29. Human-level control through deep reinforcement learning10.1038/nature14236 ↗
30. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2016, January 2–4). Continuous control with deep reinforcement learning. Proceedings of the 4th International Conference on Learning Representations (ICLR 2016), San Juan, Puerto Rico.
31. Automatic Gain Tuning Method of a Quad-Rotor Geometric Attitude Controller Using A3C10.1007/s42405-019-00233-x ↗
32. Laud, A.D. (2004). Theory and Application of Reward Shaping in Reinforcement Learning. [Ph.D. Dissertation, University of Illinois].
33. Belief Reward Shaping in Reinforcement Learning
34. Clark, J., and Amodei, D. (2022, November 09). Faulty Reward Functions in the Wild. Available online: https://openai.com/blog/faulty-reward-functions/.
35. Irpan, A. (2022, November 09). Deep Reinforcement Learning Doesn’t Work Yet. Available online: https://www.alexirpan.com/2018/02/14/rl-hard.html.
36. Exploration in deep reinforcement learning: A survey10.1016/j.inffus.2022.03.003 ↗
37. Controller exploitation-exploration reinforcement learning architecture for computing near-optimal policies10.1007/s00500-018-3225-7 ↗
38. Schäfer, L., Christianos, F., Hanna, J., and Albrecht, S.V. (2021). Decoupling exploration and exploitation in reinforcement learning. ArXiv.
39. Perspective view of autonomous control in unknown environment: Dual control for exploitation and exploration vs reinforcement learning10.1016/j.neucom.2022.04.131 ↗
40. How to conduct a bibliometric analysis: An overview and guidelines10.1016/j.jbusres.2021.04.070 ↗
41. Citation network analysis10.1080/1750984X.2021.1989705 ↗
42. A literature review on accessibility using bibliometric analysis techniques10.1016/j.jtrangeo.2020.102810 ↗
43. Bibliometrix: An R-tool for comprehensive science mapping analysis10.1016/j.joi.2017.08.007 ↗
44. (2022, November 01). R-Project. Available online: https://www.r-project.org.
45. (2022, November 01). Cytoscape. Available online: https://cytoscape.org.
46. Metabolomics and pesticides: Systematic literature review using graph theory for analysis of references10.22490/24629448.1735 ↗
47. Thakur, D., Wang, J., and Cozzens, S. (2011, January 15–17). What does international co-authorship measure?. Proceedings of the 2011 Atlanta Conference on Science and Innovation Policy, Atlanta, GA, USA.10.1109/ACSIP.2011.6064489 ↗
48. Influence of international co-authorship on the research citation impact of young universities10.1007/s11192-016-1905-6 ↗
49. Does Bradford’s Law of Scattering predict the size of the literature in Cochrane Reviews?10.3163/1536-5050.100.2.013 ↗
50. Gu, S., Holly, E., Lillicrap, T., and Levine, S. (June, January 29). Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates. Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore.10.1109/ICRA.2017.7989385 ↗
51. End-to-End Training of Deep Visuomotor Policies
52. On the Theory of the Brownian Motion10.1103/PhysRev.36.823 ↗
53. Obstacle Avoidance for Kinematically Redundant Manipulators in Dynamically Varying Environments10.1177/027836498500400308 ↗
54. An Introduction to Deep Reinforcement Learning10.1561/2200000071 ↗
55. Chen, L. (2021). Deep reinforcement learning. Deep Learning and Practice with MindSpore, Springer.10.1007/978-981-16-2233-5 ↗
56. Deep reinforcement learning: A brief survey10.1109/MSP.2017.2743240 ↗
57. Sewak, M. (2019). Deep Reinforcement Learning—Frontiers of Artificial Intelligence, Springer. [1st ed.].10.1007/978-981-13-8285-7 ↗
58. State-of-the-Art Reinforcement Learning Algorithms
59. Moerland, T.M., Broekens, J., and Jonker, C.M. (2020). Model-based Reinforcement Learning: A Survey. arXiv.
60. Kaiser, Ł., Babaeizadeh, M., Miłos, P., Osinski, B., Campbell, R.H., Czechowski, K., Erhan, D., Finn, C., Kozakowski, P., and Levine, S. (2020, January 30). Model-Based Reinforcement Learning for Atari. Proceedings of the International Conference on Learning Representations (ICLR 2020), Addis Ababa, Ethiopia.
61. Plaat, A., Kosters, W., and Preuss, M. (2020). Deep model-based reinforcement learning for high-dimensional problems, a survey. arXiv.
62. Janner, M., Fu, J., Zhang, M., and Levine, S. (2019, January 8–14). When to trust your model: Model-based policy optimization. Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada.
63. Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P., and Ba, J. (2019). Benchmarking model-based reinforcement learning. arXiv.
64. Sun, W., Jiang, N., Krishnamurthy, A., Agarwal, A., and Langford, J. (2019, January 25–28). Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. Proceedings of the Thirty-Second Conference on Learning Theory, Phoenix, AZ, USA.
65. Luo, F.-M., Xu, T., Lai, H., Chen, X.-H., Zhang, W., and Yu, Y. (2022). A survey on model-based reinforcement learning. arXiv.
66. Mastering the game of Go without human knowledge10.1038/nature24270 ↗
67. Feng, D., Gomes, C.P., and Selman, B. (2021, January 7–15). Solving Hard AI Planning Instances Using Curriculum-Driven Deep Reinforcement Learning. Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020), Yokohama, Japan.10.24963/ijcai.2020/304 ↗
68. A robot arm digital twin utilising reinforcement learning10.1016/j.cag.2021.01.011 ↗
69. A digital twin to train deep reinforcement learning agent for smart manufacturing plants: Environment, interfaces and intelligence10.1016/j.jmsy.2020.06.012 ↗
70. Wiering, M.A., Withagen, M., and Drugan, M.M. (2014, January 9–12). Model-based multi-objective reinforcement learning. Proceedings of the 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), Orlando, FL, USA.10.1109/ADPRL.2014.7010622 ↗
71. Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. (May, January 30). METRPO: Model-ensemble trust-region policy optimization. Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada.
72. Rajeswaran, A., Mordatch, I., and Kumar, V. (2020, January 13–18). A game theoretic framework for model based reinforcement learning. Proceedings of the 37th International Conference on Machine Learning (ICML 2020), Virtual.
73. Model-based policy optimization with unsupervised model adaptation
74. Ha, D., and Schmidhuber, J. (2018). World Models. Forecast. Bus. Econ., 201–209.
75. Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Rezende, D.J., Badia, A.P., Vinyals, O., Heess, N., and Li, Y. (2017, January 4–9). Imagination-Augmented Agents for Deep Reinforcement Learning. Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA.
76. Edwards, A.D., Downs, L., and Davidson, J.C. (2018). Forward-backward reinforcement learning. arXiv.
77. van Hasselt, H., Hessel, M., and Aslanides, J. (2019, January 8–14). When to use parametric models in reinforcement learning?. Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, BC, Canada.
78. Model-free reinforcement learning from expert demonstrations: A survey10.1007/s10462-021-10085-1 ↗
79. Otto, F. (2021). Model-Free Deep Reinforcement Learning—Algorithms and Applications. Reinforcement Learning Algorithms: Analysis and Applications, Springer.10.1007/978-3-030-41188-6_10 ↗
80. Hausknecht, M., Stone, P., and Mc, O. (2016, January 9–15). On-Policy vs. Off-Policy Updates for Deep Reinforcement Learning. Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI 2016), New York City, NY, USA.
81. Tan, Z., and Karakose, M. (2021). On-Policy Deep Reinforcement Learning Approach to Multi Agent Problems. Interdisciplinary Research in Technology and Management, CRC Press.10.1201/9781003202240-58 ↗
82. Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., and Michalski, M. (2020). What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv.
83. Agarwal, R., Schuurmans, D., and Norouzi, M. (2022, October 14). Striving for Simplicity in Off-Policy Deep Reinforcement Learning. Available online: https://openreview.net/forum?id=ryeUg0VFwr.
84. Zimmer, M., Boniface, Y., and Dutech, A. (2016, January 5–10). Off-Policy Neural Fitted Actor-Critic. Proceedings of the Deep Reinforcement Learning Workshop (NIPS 2016), Barcelona, Spain.
85. Fujimoto, S., Meger, D., and Precup, D. (2019, January 9–15). Off-policy deep reinforcement learning without exploration. Proceedings of the 36th International Conference on Machine Learning (ICML 2019), Long Beach, CA, USA.
86. Clemente, A.V., Castejón, H.N., and Chandra, A. (2017). Efficient Parallel Methods for Deep Reinforcement Learning. arXiv.
87. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv.
88. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Openai, O.K. (2017). Proximal Policy Optimization Algorithms. arXiv.
89. Dong, H., Ding, Z., and Zhang, S. (2020). Deep Q-networks. Deep Reinforcement Learning: Fundamentals, Research and Applications, Springer Nature.10.1007/978-981-15-4095-0 ↗
90. Dabney, W., Rowland, M., Bellemare, M.G., and Munos, R. (2018, January 2–7). Distributional Reinforcement Learning with Quantile Regression. Proceedings of the 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, Hilton New Orleans Riverside, New Orleans, LA, USA.10.1609/aaai.v32i1.11791 ↗
91. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W. (2022, October 04). Hindsight Experience Replay. Available online: https://goo.gl/SMrQnI.
92. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2022, October 04). Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. January 2018. Available online: http://arxiv.org/abs/1801.01290.
93. Casas, N. (2017). Deep deterministic policy gradient for urban traffic light control. arXiv.
94. Fujimoto, S., van Hoof, H., and Meger, D. (2018, January 10–15). Addressing Function Approximation Error in Actor-Critic Methods. Proceedings of the 35th International Conference on Machine Learning (ICML 2018), Stockholm, Sweden.
95. Saeed, M., Nagdi, M., Rosman, B., and Ali, H.H.S.M. (March, January 26). Deep Reinforcement Learning for Robotic Hand Manipulation. Proceedings of the 2020 International Conference on Computer, Control, Electrical, and Electronics Engineering (ICCCEEE 2020), Khartoum, Sudan.10.1109/ICCCEEE49695.2021.9429619 ↗
96. Smart manufacturing scheduling: A literature review10.1016/j.jmsy.2021.09.011 ↗
97. Manufacturing process control through integration of neural networks and fuzzy model10.1016/S0165-0114(96)00382-X ↗
98. Artificial intelligence in manufacturing and logistics systems: Algorithms, applications, and case studies10.1080/00207543.2020.1752488 ↗
99. Industry 4.0 smart reconfigurable manufacturing machines10.1016/j.jmsy.2021.03.001 ↗
100. Reinforcement learning for facilitating human-robot-interaction in manufacturing10.1016/j.jmsy.2020.06.018 ↗