A digital twin-driven dynamic path planning approach for multiple automatic guided vehicles based on deep reinforcement learning
Abstract
With the increasing demand for customization, the tendency of mechanical manufacturing has gradually shifted to flexible and mixed-line production, which brings new challenges to the existing scheduling pattern. As an indispensable part, logistics is responsible for establishing connections among various production equipment and processes. Meanwhile, the promotion of digital twin theory introduces an application schema for the logistics system. However, there is still a deficiency in the real-time dispatching and path planning of logistics equipment due to the uncontrollability of algorithm efficiency for complex scenes. To fill this gap, a digital twin-driven dynamic path planning approach for multiple automatic guided vehicles (AGVs) is proposed. Firstly, the AGVs are virtualized as the major component of logistics systems, while the ontology expression of logistics tasks is consistently accomplished as well. Secondly, the digital twin-driven application framework of multi-AGV dispatching is established. Moreover, a dynamic path planning method for AGVs relying on deep reinforcement learning is implemented. A segmented path planning method is illustrated considering potential route conflicts, which is regarded as the key contribution of the presented research. At last, a case study is illustrated to show the entire process of multiple vehicle path planning and conflict resolution.
Get full access to this article
View all access and purchase options for this article.
References
1. Sony M, Antony J, Mc Dermott O, et al. An empirical examination of benefits, challenges, and critical success factors of Industry 4.0 in manufacturing and service sector. Technol Soc 2021; 67: 101754.
2. Garza-Reyes JA. Lean and green – a systematic review of the state of the art literature. J Clean Prod 2015; 102: 18–29.
3. Gunasekaran A, Yusuf YY, Adeleye EO, et al. Agile manufacturing: an evolutionary review of practices. Int J Prod Res 2019; 57(15–16): 5154–5174.
4. Wang B, Zheng P, Yin Y, et al. Toward human-centric smart manufacturing: a human-cyber-physical systems (HCPS) perspective. J Manuf Syst 2022; 63: 471–490.
5. Xue G, Xia Y. A flexible logistics system for intelligent manufacturing workshop. In: 2019 IEEE 10th international conference on mechanical and intelligent manufacturing technologies (ICMIMT), Cape Town, South Africa, 15–17 February 2019, pp.133–137. New York: IEEE.
6. Zhang Y, Guo Z, Lv J, et al. A framework for smart production-logistics systems based on CPS and industrial IoT. IEEE Trans Ind Inform 2018; 14(9): 4019–4032.
7. Tetik M, Peltokorpi A, Seppänen O, et al. Kitting logistics solution for improving on-site work performance in construction projects. J Constr Eng Manag 2021; 147(1): 05020020.
8. Caputo AC, Pelagagge PM, Salini P. A model for planning and economic comparison of manual and automated kitting systems. Int J Prod Res 2021; 59(3): 885–908.
9. Zhou L, Zhang L, Horn BK. Collaborative optimization for logistics and processing services in cloud manufacturing. Robot Comput Integr Manuf 2021; 68: 102094.
10. Grieves M, Vickers J. Digital twin: mitigating unpredictable, undesirable emergent behavior in complex systems. In: Kahlen J, Flumerfelt S, Alves A (eds) Transdisciplinary perspectives on complex systems. Cham: Springer, 2017, pp.85–113.
11. Semeraro C, Lezoche M, Panetto H, et al. Digital twin paradigm: a systematic literature review. Comput Ind 2021; 130: 103469.
12. Liu M, Fang S, Dong H, et al. Review of digital twin about concepts, technologies, and industrial applications. J Manuf Syst 2021; 58: 346–361.
13. Leng J, Wang D, Shen W, et al. Digital twins-based smart manufacturing system design in Industry 4.0: a review. J Manuf Syst 2021; 60: 119–137.
14. Zhang H, Qi Q, Tao F. A multi-scale modeling method for digital twin shop-floor. J Manuf Syst 2022; 62: 417–428.
15. Zhang L, Guo Y, Qian W, et al. Modelling and online training method for digital twin workshop. Int J Prod Res 2022; 61: 3943–3962.
16. Huang Z, Shen Y, Li J, et al. A survey on AI-driven digital twins in Industry 4.0: smart manufacturing and advanced robotics. Sensors 2021; 21(19): 6340.
17. Kherbache M, Maimour M, Rondeau E. When digital twin meets network softwarization in the industrial IoT: real-time requirements case study. Sensors 2021; 21(24): 8194.
18. Park KT, Son YH, Noh SD. The architectural framework of a cyber physical logistics system for digital-twin-based supply chain control. Int J Prod Res 2021; 59(19): 5721–5742.
19. Pan YH, Qu T, Wu NQ, et al. Digital twin based real-time production logistics synchronization system in a multi-level computing architecture. J Manuf Syst 2021; 58: 246–260.
20. Hofmann E, Rüsch M. Industry 4.0 and the current status as well as future prospects on logistics. Comput Ind 2017; 89: 23–34.
21. Ray PP. A survey on internet of things architectures. J King Saud Univ - Comput Inf Sci 2018; 30(3): 291–319.
22. Sisinni E, Saifullah A, Han S, et al. Industrial internet of things: challenges, opportunities, and directions. IEEE Trans Ind Inform 2018; 14(11): 4724–4734.
23. Guo D, Zhong RY, Rong Y, et al. Synchronization of shop-floor logistics and manufacturing under IIoT and digital twin-enabled graduation intelligent manufacturing system. IEEE Trans Cybern 2023; 53: 2005–2016.
24. Korth B, Schwede C, Zajac M. Simulation-ready digital twin for realtime management of logistics systems. In: 2018 IEEE international conference on big data (big data), Seattle, WA, USA, 10–13 December 2018, pp.4194–4201. New York: IEEE.
25. Coelho F, Relvas S, Barbosa-Póvoa AP. Simulation-based decision support tool for in-house logistics: the basis for a digital twin. Comput Ind Eng 2021; 153: 107094.
26. Wang G, Gunasekaran A, Ngai EWT, et al. Big data analytics in logistics and supply chain management: certain investigations for research and applications. Int J Prod Econ 2016; 176: 98–110.
27. Zhao XF, Liu HZ, Lin SX, et al. Design and implementation of a multiple AGV scheduling algorithm for a job-shop. Int J Simul Model 2020; 19(1): 134–145.
28. Zhou L, Zhang L, Ren L. Modelling and simulation of logistics service selection in cloud manufacturing. Procedia CIRP 2018; 72: 916–921.
29. Aghamohammadzadeh E, Malek M, Valilai OF. A novel model for optimisation of logistics and manufacturing operation service composition in cloud manufacturing system focusing on cloud-entropy. Int J Prod Res 2020; 58(7): 1987–2015.
30. Zhong M, Yang Y, Dessouky Y, et al. Multi-AGV scheduling for conflict-free path planning in automated container terminals. Comput Ind Eng 2020; 142: 106371.
31. Duchoň F, Babinec A, Kajan M, et al. Path planning with modified a star algorithm for a mobile robot. Procedia Eng 2014; 96: 59–69.
32. Lin M, Yuan K, Shi C, et al. Path planning of mobile robot based on improved A* algorithm. In: 2017 Chinese control and decision conference, Chongqing, China, 28–30 May 2017, pp.3622–3628. New York: IEEE.
33. Hu YJ, Dong LC, Xu L. Multi-AGV dispatching and routing problem based on a three-stage decomposition method. Math Biosci Eng 2020; 17(5): 5150–5172.
34. Lian Y, Yang Q, Xie W, et al. Cyber-physical system-based heuristic planning and scheduling method for multiple automatic guided vehicles in logistics systems. IEEE Trans Ind Inform 2021; 17(11): 7882–7893.
35. Tao Q, Sang H, Guo H, et al. Improved particle swarm optimization algorithm for AGV path planning. IEEE Access 2021; 9: 33522–33531.
36. Farooq B, Bao J, Raza H, et al. Flow-shop path planning for multi-automated guided vehicles in intelligent textile spinning cyber-physical production systems dynamic environment. J Manuf Syst 2021; 59: 98–116.
37. Chen J, Zhang X, Peng X, et al. Efficient routing for multi-AGV based on optimized ant-agent. Comput Ind Eng 2022; 167: 108042.
38. Kober J, Bagnell JA, Peters J. Reinforcement learning in robotics: a survey. Int Journal Robotics Res 2013; 32(11): 1238–1274.
39. Zhang L, Yan Y, Hu Y, et al. Reinforcement learning and digital twin-based real-time scheduling method in intelligent manufacturing systems. IFAC-PapersOnLine 2022; 55(10): 359–364.
40. Zheng P, Xia L, Li C, et al. Towards self-x cognitive manufacturing network: an industrial knowledge graph-based multi-agent reinforcement learning approach. J Manuf Syst 2021; 61: 16–26.
41. Li C, Zheng P, Yin Y, et al. Deep reinforcement learning in smart manufacturing: a review and prospects. CIRP J Manuf Sci Technol 2023; 40: 75–101.
42. Hu H, Jia X, He Q, et al. Deep reinforcement learning based AGVs real-time scheduling with mixed rule for flexible shop floor in Industry 4.0. Comput Ind Eng 2020; 149: 106749.
43. Mueller-Zhang Z, Oliveira Antonino P, Kuhn T. Integrated planning and scheduling for customized production using digital twins and reinforcement learning. IFAC-PapersOnLine 2021; 54(1): 408–413.
44. Li C, Zheng P, Yin Y, et al. An ar-assisted deep reinforcement learning-based approach towards mutual-cognitive safe human-robot interaction. Robot Comput Integr Manuf 2023; 80: 102471.
45. Hu H, Yang X, Xiao S, et al. Anti-conflict AGV path planning in automated container terminals based on multi-agent reinforcement learning. Int J Prod Res 2023; 61(65): 65–80.
46. Wei Q, Yan Y, Zhang J, et al. A self-attention-based deep reinforcement learning approach for AGV dispatching systems. IEEE Trans Neural Netw Learn Syst 2022; 1–12.
47. Bao J, Guo D, Li J, et al. The modelling and operations for the digital twin in the context of manufacturing. Enterp Inf Syst 2019; 13(4): 534–556.
48. Bao Q, Zhao G, Yu Y, et al. The ontology-based modeling and evolution of digital twin for assembly workshop. Int J Adv Manuf Technol 2021; 117(1): 395–411.
49. Bao Q, Zhao G, Yu Y, et al. Ontology-based modeling of part digital twin oriented to assembly. Proc IMechE, Part B: J Engineering Manufacture 2022; 236(1–2): 16–28.
50. Mnih V, Kavukcuoglu K, Silver D, et al. Playing Atari with deep reinforcement learning. arXiv preprint arXiv 13125602, 2013.
51. Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature 2015; 518(7540): 529–533.
52. De Ryck M, Versteyhe M, Debrouwere F. Automated guided vehicle systems, state-of-the-art control algorithms and techniques. J Manuf Syst 2020; 54: 152–173.
Cite article
Cite article
Cite article
OR
Download to reference manager
If you have citation software installed, you can download article citation data to the citation manager of your choice
Information, rights and permissions
Information
Published In
Article first published online: June 6, 2023
Issue published: March 2024
Keywords
Authors
Metrics and citations
Metrics
Journals metrics
This article was published in Proceedings of the Institution of Mechanical Engineers, Part B: Journal of Engineering Manufacture.
View All Journal MetricsArticle usage*
Total views and downloads: 248
*Article usage tracking started in December 2016
Altmetric
See the impact this article is making through the number of times it’s been read, and the Altmetric Score.
Learn more about the Altmetric Scores
Articles citing this one
Receive email alerts when this article is cited
Web of Science: 1 view articles Opens in new tab
Crossref: 5
- A Bibliometric Analysis on Artificial Intelligence in the Production Process of Small and Medium Enterprises
- A Systematic Review on Reinforcement Learning for Industrial Combinatorial Optimization Problems
- Advances in Production Management Systems. Production Management Systems for Volatile, Uncertain, Complex, and Ambiguous Environments
- 2024 5th International Conference on Computer Engineering and Application (ICCEA)
- Path planning method for Camellia oleifera forest trenching operation based on human-robot collaboration
Figures and tables
Figures & Media
Tables
Get access
Get access
Access options
If you have access to journal content via a personal subscription, university, library, employer or society, select from the options below:
I am signed in as:
View my profileSign out
I can access personal subscriptions, purchases, paired institutional access and free tools such as favourite journals, email alerts and saved searches.
Access journal content via a university, library or employer subscription.
IOM3 members can access this journal content using society membership credentials.
Alternatively, view purchase options below:
Purchase 24 hour online access to view and download content.
Access journal content via a DeepDyve subscription or find out more about this option.
