Q-learning-based genetic optimization rescheduling model for steelmaking-continuous casting section with order disturbance

CHEN Hongzhi; SHAO Xin; LIU Qing; ZHANG Jiangshan; GAO Shan

doi:10.13228/j.boyuan.issn0449-749x.20230403

PDF(3063 KB)

Iron and Steel ›› 2023, Vol. 58 ›› Issue (11) : 90-99. DOI: 10.13228/j.boyuan.issn0449-749x.20230403

Q-learning-based genetic optimization rescheduling model for steelmaking-continuous casting section with order disturbance

Author information +

History +

Abstract

The demand for steel products in today′s market is characterized by multiple varieties, small batches, multiple specifications, and high quality, resulting in complex orders for steelmaking plants. The use of different steel grade continuous casting technology can effectively increase the number of continuous casting furnaces, reduce the number of castings required to complete contracts, and improve production continuity. Driven by production orders, the steelmaking-continuous casting section often interferes with the original production plan due to the emergence of emergency orders. This article develops a response strategy based on the matching degree between the steel grades of emergency orders and the required production process with the actual production situation of one steelmaking plants, such as production capacity, production process, product inventory, and raw material inventory. A rescheduling model for steelmaking-continuous casting section is established with the optimization objectives of minimum total waiting time, minimum total delaying time, and minimum completion time in the production plan. An improved genetic algorithm based on Q-learning is proposed for solution. This study conducted simulation experiments using two typical production modes in a certain steelmaking plant, which are different steel grade sequence casting laminar production mode and 4BOF-3CCM turbulent production mode producing the same steel grade. The results showed that the established model can effectively solve the problem of orders disturbance rescheduling, reduce the increase in total waiting time in the production plan caused by emergency orders, and reduce the delaying time of the production plan. This article tests the performance of Q-learning based genetic algorithms by solving the optimal sorting problem. Compared to classical genetic algorithms, Q-learning based genetic algorithms can find the optimal solution that is more in line with the optimization objective, with fewer iterations to obtain the optimal solution. Compared to self-adaptive genetic algorithms, Q-learning based genetic algorithms reduce the running time by 95.37%, The above results indicate that the genetic algorithm based on Q-learning improvement has good solving performance.

Key words

steelmaking-continuous casting / production scheduling / order disturbance / Q-learning / genetic algorithm / sequence casting of different steel grades

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations

CHEN Hongzhi, SHAO Xin, LIU Qing, et al. Q-learning-based genetic optimization rescheduling model for steelmaking-continuous casting section with order disturbance[J]. Iron and Steel, 2023, 58(11): 90-99 https://doi.org/10.13228/j.boyuan.issn0449-749x.20230403

References

[1] 殷瑞钰. 流程型制造流程的本质与共性规律[J]. 钢铁, 2023, 58(2): 1. (YIN R Y. Essence and common law of a process-oriented manufacturing process[J]. Iron Steel, 2023, 58(2): 1.)
[2] YIN R Y. Metallurgical process engineering[M/OL]. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011[2023-07-02]. https://link.springer.com/10.1007/978-3-642-13956-7.
[3] 邵鑫, 杨建平, 王柏琳, 等. 炼钢-连铸区段多工序运行协同控制[J]. 钢铁, 2021, 56(8): 101.(SHAO X, YANG J P, WANG B L, et al. Cooperative control of multi-process operation in steelmaking-continuous casting section[J]. Iron Steel, 2021, 56(8): 101.)
[4] 刘青, 刘倩, 杨建平, 等. 炼钢-连铸生产调度的研究进展[J]. 工程科学学报, 2020, 42(2): 144. (LIU Q, LIU Q, YANG J P, et al. Process of research on steelmaking-continuous casting production scheduling[J]. Chinese Journal of Engineering, 2020, 42(2): 144.)
[5] 刘青, 邵鑫, 杨建平, 等. 炼钢厂多尺度建模与协同制造[J]. 工程科学学报, 2021, 43(12): 1698. (LIU Q, SHAO X, YANG J P, et al. Multiscale modeling and collaborative manufacturing for steelmaking plants[J]. Chinese Journal of Engineering, 2021, 43(12):1698.)
[6] 刘青,王彬,汪宙,等.高品质钢炼钢-连铸过程的精益制造[J]. 连铸, 2016(3): 1. (LIU Q, WANG B, WANG Z, et al. Fine manufacturing for steelmaking-continuous casting process of high quality steel[J]. Continuous Casting, 2016(3): 1.)
[7] 张川, 王皓, 张沅, 等. 连铸机异钢种混合浇铸生产实践[J]. 钢铁研究, 2015, 43(4): 53. (ZHANG C, WANG H, ZHANG Y, et al. Production practice on steel grade transition of continuous casting[J]. Research on Iron and Steel, 2015, 43(4): 53.)
[8] 董金刚, 陈向东, 章远杰, 等. 提高中间包连浇炉数的实践[J]. 连铸, 2017, 42(6): 5. (DONG J G, CHEN X D, ZHANG Y J, et al. Practice of improving tundish continuous casting number[J]. Continuous Casting, 2017, 42(6): 5.)
[9] 董金刚.宝钢提高板坯连铸CAST炉数的措施[J]. 连铸, 2006(4): 16. (DONG J G. Measure of increasing CAST heat number for slab continuous casting in Baosteel[J]. Continuous Casting, 2006(4): 16.)
[10] 董金刚,余茂华,唐立新.提高连铸生产能力的措施与效果[J]. 中国冶金, 2007(9): 14. (DONG J G, YU M H, TANG L X. Measures and effects on increasing continuous casting productivity[J]. China Metallurgy, 2007(9): 14.)
[11] 李东明, 董方, 付岳. 优化异钢种连续浇注的生产实践[J]. 包钢科技, 2016, 42(6): 28. (LI D M, DONG F, FU Y. Production practice of optimizing different steel grade continuous casting[J]. Science and Technology of Baotou Steel, 2016, 42(6): 28.)
[12] 金焰, 王秀英. 改进的差分进化算法求解炼钢-连铸动态调度问题[J]. 计算机测量与控制, 2022, 30(6): 272. (JIN Y, WANG X Y. Improved different evolution algorithm to solve steelmaking-continuous casting dynamic scheduling problem[J]. Computer Measurement and Control, 2022, 30(6): 272.)
[13] 郭煜. 急件订单插入下混流生产系统计划调度问题研究[D]. 赣州:江西理工大学, 2021. (GUO Y. Dynamic Scheduling Problem of Mixed-flow Production System Considering Urgent Order Interference[D]. Ganzhou:Jiangxi University of Science and Technology, 2021.)
[14] 王晶, 郑亚楠. 考虑机器故障的炼钢连铸重调度模型与算法设计[J]. 冶金自动化, 2015, 39(2): 28. (WANG J, ZHENG Y L. Model and algorithm design on rescheduling of steelmaking and continuous casting[J]. Metallurgical Industry Automation, 2015, 39(2): 28.)
[15] 侯东亮, 李铁克. 考虑新任务插入的炼钢-连铸重调度模型与算法[J]. 工业工程, 2012, 15(5): 33. (HOU D L, LI T K. Rescheduling model and algorithm for stellmaking and continuous casting with new tasks arrival[J]. Industrial Engineering Journal, 2012, 15(5): 33.)
[16] 朱赟海, 罗陆锋, 卢清华, 等. 面向订单扰动的柔性作业车间动态调度方法[J]. 计算机集成制造系统: 1. http://kns.cnki.net/kcms/detail/11.5946.TP.20230223.1654.036.html. (ZHUN Y H, LUO L F, LU Q H, et al. Flexible job shop dynamic scheduling method for order disturbance[J]. Computer Integrated Manufacturing Systems:1. http://kns.cnki.net/kcms/detail/11.5946.TP.20230223.1654.036.html.)
[17] 王晶, 郑亚楠. 基于约束的炼钢连铸动态调度模型与算法设计[J]. 制造业自动化, 2014, 36(20): 110.(WANG J, ZHENG Y L. Model and algorithm design dynamic scheduling of steelmaking and continuous casting based on constraint[J]. Manufacturing Automation, 2014, 36(20): 10.)
[18] YANG J P, WANG B L, LIU Q, et al. Scheduling model for the practical steelmaking-continuous casting production and heuristic algorithm based on the optimization of "Furnace-caster Matching" Mode[J]. ISIJ International, 2020, 60(6): 1213.
[19] GARCIA-MENENDEZ D, MORAN-PALACIOS H, ORTEGA-FERNANDEZ F, et al. Scheduling in continuous steelmaking casting: A systematic review[J]. ISIJ International, 2020, 60(6): 1097.
[20] LIU C, TANG L, ZHAO C. A novel dynamic operation optimization method based on multiobjective deep reinforcement learning for steelmaking process[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023: 1.
[21] PANG X, GAO L, PAN Q, et al. A novel lagrangian relaxation level approach for scheduling steelmaking-refining-continuous casting production[J]. Journal of Central South University, 2017, 24(2): 467.
[22] MAO K, PAN Q, PANG X, et al. An effective lagrangian relaxation approach for rescheduling a steelmaking-continuous casting process[J]. Control Engineering Practice, 2014, 30: 67.
[23] DAI M, TANG D, GIRET A, et al. Energy-efficient scheduling for a flexible flow shop using an improved genetic-simulated annealing algorithm[J]. Robotics and Computer-Integrated Manufacturing, 2013, 29(5): 418.
[24] PACCIARELLI D, PRANZO M. Production scheduling in a steelmaking-continuous casting plant[J]. Computers and Chemical Engineering, 2004, 28(12):2823.
[25] 董胡适. 基于遗传编程的炼钢-连铸调度知识系统研究[D]. 武汉:武汉科技大学, 2019. (DONG H S. Research on Konwledge System of Steelmaking-continuous Casting Scheduling Based on Genetic Programming[D]. Wuhan:Wuhan University of Science and Technology, 2019.)
[26] 任子武, 伞冶. 自适应遗传算法的改进及在系统辨识中应用研究[J]. 系统仿真学报, 2006(1): 41. (REN Z W, SAN Y. Improved adaptive genetic algorithm and its application research in parameter identification[J]. Journal of System Simulation, 2006(1): 41.)
[27] 刘倩, 杨建平, 王柏琳, 等. 基于“炉-机对应”的炼钢-连铸生产调度问题遗传优化模型[J]. 工程科学学报, 2020, 42(5): 645. (LIU Q, YANG J P, WANG B L, et al. Genetic optimization model of steelmaking-continuous casting production scheduling based on the "furnace-caster coordinating" strategy[J]. Chinese Journal of Engineering, 2020, 42(5): 645.)
[28] 李润佛, 张新宇, 李俊杰, 等. 基于强化学习的自学习遗传算法在船舶调度中的应用[J]. 大连海事大学学报, 2022, 48(3): 20. (LI R F, ZHANG X Y, LI J J, et al. Application of self-learning genetic algorithm based on reinforcement learning in ship scheduling[J]. Journal of Dalian Maritime University, 2022, 48(3): 20.)
[29] 王玉芳, 严洪森. 基于改进Q学习的知识化制造自适应动态调度策略[J]. 控制与决策, 2015, 30(11): 1930. (WANG Y F, YAN H S. Adaptive dynamic scheduling strategy in konwledgeable manufacturing based on improved Q-learning[J]. Control and Decision, 2015, 30(11): 1930.)
[30] 王艳红, 尹涛, 谭园园, 等. 基于规则与Q学习的作业车间动态调度算法[J]. 计算机集成制造系统,2023. http://kns.cnki.net/kcms/detail/11.5946.TP.20230506.1406.002.html. (WANG Y H, YIN T, TAN Y Y, et al. Dispatching rule and Q-learning based dynamic job shop scheduling algorithm[J]. Computer Integrated Manufacturing Systems,2023. http://kns.cnki.net/kcms/detail/11.5946.TP.20230506.1406.002.html.)
[31] 唐雨婷, 李雪峰, 张金学, 等. 基于深度学习和Q学习的机械臂功能设计与实现[J]. 长江信息通信, 2023, 36(2): 97. (TANG Y T, LI X F, ZHANG J X, et al. Function design and implementation of robot arm based on deep learning and Q-learning[J]. Changjiang Information and Communications, 2023 36(2): 97.)
[32] 王维祺, 叶春明, 谭晓军. 基于Q学习算法的作业车间动态调度[J]. 计算机系统应用, 2020, 29(11): 218. (WANG W Q, YE C M, TAN X J. Job shop dynamic scheduling based on Q-learning algorithm[J]. Computer Systems and Applications, 2020, 29(11): 218.)
[33] JANG B, KIM M, HARERIMANA G, et al. Q-learning algorithms: A comprehensive classification and applications[J]. IEEE Access, 2019(7): 133653.
[34] LONG J, ZHENG Z, GAO X. Dynamic scheduling in steelmaking-continuous casting production for continuous caster breakdown[J]. International Journal of Production Research, 2017, 55(11): 3197.
[35] 刘青, 黄星武, 富平原. 炼钢厂系统生产模式优化[J]. 北京科技大学学报, 2005(6): 736. (LIU Q, HUANG X W, FU P Y. Production mode optimization of a steelmaking workshop system[J]. Journal of University of Science and Technology Beijing, 2005(6): 736.)