Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

doi:10.1016/j.cjche.2023.12.005

Chinese Journal of Chemical Engineering ›› 2024, Vol. 69 ›› Issue (5): 63-71.DOI: 10.1016/j.cjche.2023.12.005

Previous Articles Next Articles

Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

Hui Tan¹, Xiaodong Hong^1,2, Zuwei Liao¹, Jingyuan Sun¹, Yao Yang¹, Jingdai Wang¹, Yongrong Yang¹

1. State Key Laboratory of Chemical Engineering, College of Chemical and Biological Engineering, Zhejiang University, Hangzhou 310027, China;
2. ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou 311215, China

Received:2023-11-11 Revised:2023-12-14 Online:2024-07-01 Published:2024-05-28
Contact: Zuwei Liao,E-mail:liaozw@zju.edu.cn
Supported by:
The financial support provided by the Project of National Natural Science Foundation of China (U22A20415, 21978256, 22308314) and “Pioneer” and “Leading Goose” Research & Development Program of Zhejiang (2022C01SA442617) are gratefully acknowledged. Besides, the authors would like to thank artificial intelligence + High Performance Computing Center of ZJU-ICI.

Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

Hui Tan¹, Xiaodong Hong^1,2, Zuwei Liao¹, Jingyuan Sun¹, Yao Yang¹, Jingdai Wang¹, Yongrong Yang¹

1. State Key Laboratory of Chemical Engineering, College of Chemical and Biological Engineering, Zhejiang University, Hangzhou 310027, China;
2. ZJU-Hangzhou Global Scientific and Technological Innovation Center, Hangzhou 311215, China

通讯作者: Zuwei Liao,E-mail:liaozw@zju.edu.cn
基金资助:
The financial support provided by the Project of National Natural Science Foundation of China (U22A20415, 21978256, 22308314) and “Pioneer” and “Leading Goose” Research & Development Program of Zhejiang (2022C01SA442617) are gratefully acknowledged. Besides, the authors would like to thank artificial intelligence + High Performance Computing Center of ZJU-ICI.

Abstract

Abstract: Heat integration is important for energy-saving in the process industry. It is linked to the persistently challenging task of optimal design of heat exchanger networks (HEN). Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem, it is not easy to find solutions of high quality for large-scale problems. The reinforcement learning (RL) method, which learns strategies through ongoing exploration and exploitation, reveals advantages in such area. However, due to the complexity of the HEN design problem, the RL method for HEN should be dedicated and designed. A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods. An insightful state representation of the HEN structure as well as a customized reward function is introduced. A Q-learning algorithm is applied to update the HEN structure using the ϵ-greedy strategy. Better results are obtained from three literature cases of different scales.

Key words: Heat exchanger network, Reinforcement learning, Mathematical programming, Process design

摘要： Heat integration is important for energy-saving in the process industry. It is linked to the persistently challenging task of optimal design of heat exchanger networks (HEN). Due to the inherent highly nonconvex nonlinear and combinatorial nature of the HEN problem, it is not easy to find solutions of high quality for large-scale problems. The reinforcement learning (RL) method, which learns strategies through ongoing exploration and exploitation, reveals advantages in such area. However, due to the complexity of the HEN design problem, the RL method for HEN should be dedicated and designed. A hybrid strategy combining RL with mathematical programming is proposed to take better advantage of both methods. An insightful state representation of the HEN structure as well as a customized reward function is introduced. A Q-learning algorithm is applied to update the HEN structure using the ϵ-greedy strategy. Better results are obtained from three literature cases of different scales.

关键词: Heat exchanger network, Reinforcement learning, Mathematical programming, Process design

Hui Tan, Xiaodong Hong, Zuwei Liao, Jingyuan Sun, Yao Yang, Jingdai Wang, Yongrong Yang. Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks[J]. Chinese Journal of Chemical Engineering, 2024, 69(5): 63-71.

References

[1] A.H. Masso, D.F. Rudd, The synthesis of system designs. II. Heuristic structuring, AlChE. J. 15(1)(1969)10-17.
[2] B. Linnhoff, Flower J.R, Synthesis of heat exchanger networks:I. Systematic generation of energy optimal networks, AlChE. J. 24(4)(1978)633-642.
[3] B. Linnhoff, S. Ahmad, Cost optimum heat exchanger networks-1. Minimum energy and capital using simple models for capital cost, Comput. Chem. Eng. 14(7)(1990)729-750.
[4] S. Ahmad, B. Linnhoff, R. Smith, Cost optimum heat exchanger networks-2. targets and design for detailed capital cost models, Comput. Chem. Eng. 14(7)(1990)751-767.
[5] R. Colberg, Area, cost and resilience targets for heat exchanger networks, Dissertation (Ph.D.), California Institute of Technology, Pasadena, USA,1989.
[6] J. Cerda, A.W. Westerberg, D. Mason, B. Linnhoff, Minimum utility usage in heat exchanger network synthesis A transportation problem, Chem. Eng. Sci. 38(3)(1983)373-387.
[7] J. Cerda, A.W. Westerburg, Synthesizing heat exchanger networks having restricted stream/stream matches using transportation problem formulations, Chem. Eng. Sci. 38(10)(1983)1723-1740.
[8] S.A. Papoulias, I.E. Grossmann, A structural optimization approach in process synthesis-III, Comput. Chem. Eng. 7(6)(1983)723-734.
[9] M. Escobar, J.O. Trierweiler, Optimal heat exchanger network synthesis:a case study comparison, Appl. Therm. Eng. 51(1-2)(2013)801-826.
[10] C.A. Floudas, A.R. Ciric, I.E. Grossmann, Automatic synthesis of optimum heat exchanger network configurations, AlChE. J. 32(2)(1986)276-290.
[11] T.F. Yee, I.E. Grossmann, Simultaneous optimization models for heat integration-II. Heat exchanger network synthesis, Comput. Chem. Eng. 14(10)(1990)1165-1184.
[12] S.Y. Kim, M. Bagajewicz, Global optimization of heat exchanger networks using a new generalized superstructure, Chem. Eng. Sci. 147(2016)30-46.
[13] K.M. Bjork, T. Westerlund, Global optimization of heat exchanger network synthesis problems with and without the isothermal mixing assumption, Comput. Chem. Eng. 26(11)(2002)1581-1593.
[14] K.F. Huang, E.M. Al-mutairi, I.A. Karimi, Heat exchanger network synthesis using a stagewise superstructure with non-isothermal mixing, Chem. Eng. Sci. 73(2012)30-43.
[15] P. Jongsuwat, U. Suriyapraphadilok, M. Bagajewicz, New heat exchanger network design model, Chem. Eng. Trans. 39(Special Issue)(2014)121-126.
[16] J.M. Zamora, I.E. Grossmann, A comprehensive global optimization approach for the synthesis of heat exchanger networks with no stream splits, Comput. Chem. Eng. 21(1997) S65-S70.
[17] C.S. Adjiman, I.P. Androulakis, C.A. Floudas, Global optimization of MINLP problems in process synthesis and design, Comput. Chem. Eng. 21(1997) S₄45-S₄50.
[18] M.L. Bergamini, I. Grossmann, N. Scenna, P. Aguirre, An improved piecewise outer-approximation algorithm for the global optimization of MINLP models involving concave and bilinear terms, Comput. Chem. Eng. 32(3)(2008)477-493.
[19] S.Y. Kim, P. Jongsuwat, U. Suriyapraphadilok, Bagajewicz M, Global optimization of heat exchanger networks. part 1:stages/substages superstructure, Ind.&Eng. Chem. Res. 56(2017)5944-5957.
[20] D.C. Faria, S.Y. Kim, M.J. Bagajewicz, Global optimization of the stage-wise superstructure model for heat exchanger networks, Ind. Eng. Chem. Res. 54(5)(2015)1595-1604.
[21] C.L. Chang, A. Peccini, Y.F. Wang, A.L.H. Costa, M.J. Bagajewicz, Globally optimal synthesis of heat exchanger networks. Part I:Minimal networks, AlChE. J. 66(7)(2020)16267.
[22] C.L. Chang, Z.W. Liao, A.L.H. Costa, M.J. Bagajewicz, Globally optimal synthesis of heat exchanger networks. Part II:Non-minimal networks, AlChE. J. 66(7)(2020)16264.
[23] C.L. Chang, Z.W. Liao, A.L.H. Costa, M.J. Bagajewicz, Globally optimal synthesis of heat exchanger networks. Part III:Non-isothermal mixing in minimal and non-minimal networks, AlChE. J. 67(11)(2021)17393.
[24] K.C. Furman, N.V. Sahinidis, Computational complexity of heat exchanger network synthesis, Comput. Chem. Eng. 25(9-10)(2001)1371-1390.
[25] M.A.S.S. Ravagnani, A.P. Silva, P.A. Arroyo, A.A. Constantino, Heat exchanger network synthesis and optimisation using genetic algorithm, Appl. Therm. Eng. 25(7)(2005)1003-1017.
[26] F.Y. Peng, G.M. Cui, Efficient simultaneous synthesis for heat exchanger network with simulated annealing algorithm, Appl. Therm. Eng. 78(2015)136-149.
[27] L.V. Pavao, C.B.B. Costa, M.A. da Silva Sa Ravagnani, L. Jimenez, Large-scale heat exchanger networks synthesis using simulated annealing and the novel rocket fireworks optimization, AlChE. J. 63(5)(2017)1582-1601.
[28] K.M. Yerramsetty, C.V.S. Murty, Synthesis of cost-optimal heat exchanger networks using differential evolution, Comput. Chem. Eng. 32(8)(2008)1861-1876.
[29] M. Dorigo, G. Di Caro, Ant colony optimization:a new meta-heuristic. Proceedings of the 1999 Congress on Evolutionary Computation-CEC99(Cat. No. 99TH8406). Washington, DC, USA. IEEE,(2002)1470-1477.
[30] H.L. Zhang, G.M. Cui, Optimal heat exchanger network synthesis based on improved cuckoo search via Levy flights, Chem. Eng. Res. Des. 134(2018)62-79.
[31] C.D. Hubbs, C. Li, N.V. Sahinidis, I.E. Grossmann, J.M. Wassick, A deep reinforcement learning approach for chemical production scheduling, Comput. Chem. Eng. 141(2020)106982.
[32] J.C. Hoskins, D.M. Himmelblau, Process control via artificial neural networks and reinforcement learning, Comput. Chem. Eng. 16(4)(1992)241-251.
[33] A. Khan, A. Lapkin, Searching for optimal process routes:a reinforcement learning approach, Comput. Chem. Eng. 141(2020)107027.
[34] L. Stops, R. Leenhouts, Q.H. Gao, A.M. Schweidtmann, Flowsheet generation through hierarchical reinforcement learning and graph neural networks, AlChE. J. 69(1)(2023)17938.
[35] R. Bellman, Dynamic programming, Science 153(3731)(1966)34-37.
[36] C.J.C.H. Watkins, P. Dayan, Technical note:q-learning, Mach. Learn. 8(3)(1992)279-292.
[37] M. Escobar, I.E. Grossmann, Mixed-Integer Nonlinear Programming Models for Optimal Simultaneous Synthesis of Heat Exchangers Network,(2010). www.minlp.org/library/problem/index.php?i=93.
[38] E. Castillo, L. Acevedo, A. Reverberi, Cleaner production of nitric acid by heat transfer optimization:a case study, Chem. Biochem. Eng. Q. 12(2018)157-165.
[39] A.P. Silva, M.A.S.S. Ravagnani, E.C. Biscaia, J.A. Caballero, Optimal heat exchanger network synthesis usingparticle swarm optimization, Optim. Eng. 11(3)(2010)459-470.
[40] L.V. Pavao, C.B.B. Costa, M.A. da Silva Sa Ravagnani, Automated heat exchanger network synthesis by using hybrid natural algorithms and parallel processing, Comput. Chem. Eng. 94(2016)370-386.
[41] A. Sorak, Z. Kravanja, Simultaneous MINLP synthesis of heat exchanger networks comprising different exchanger types, Comput. Aided Chem. Eng. 9(2001)1095-1100.
[42] Y. Xiao, G.M. Cui, A novel Random Walk algorithm with Compulsive Evolution for heat exchanger network synthesis, Appl. Therm. Eng. 115(2017)1118-1127.

Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

Combining reinforcement learning with mathematical programming: An approach for optimal design of heat exchanger networks

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Chun Deng, Xuantong Lu, Qixin Zhang, Jian Liu, Jui-Yuan Lee, Xiao Feng. Fuzzy optimization design of multicomponent refinery hydrogen network [J]. Chinese Journal of Chemical Engineering, 2022, 48(8): 125-139.
[2]	Siwen Gu, Lei Zhang, Yu Zhuang, Weida Li, Jian Du, Cheng Shao. Two-tier control structure design methodology applied to heat exchanger networks [J]. Chinese Journal of Chemical Engineering, 2022, 47(7): 231-244.
[3]	Lijing Zang, Kejin Huang, Yang Yuan, Xing Qian, Liang Zhang, Haisheng Chen, Shaofeng Wang. Vapor recompressed dividing-wall distillation columns: Structure and performance [J]. Chinese Journal of Chemical Engineering, 2020, 28(7): 1891-1897.
[4]	Yu Zhuang, Rui Yang, Lei Zhang, Jian Du, Shengqiang Shen. Simultaneous synthesis of sub and above-ambient heat exchanger networks including expansion process based on an enhanced superstructure model [J]. Chinese Journal of Chemical Engineering, 2020, 28(5): 1344-1356.
[5]	Siwen Gu, Linlin Liu, Lei Zhang, Yiyuan Bai, Shaojing Wang, Jian Du. Heat exchanger network synthesis integrated with flexibility and controllability [J]. Chinese Journal of Chemical Engineering, 2019, 27(7): 1474-1484.
[6]	Lixia Kang, Yongzhong Liu. Synthesis of flexible heat exchanger networks: A review [J]. Chinese Journal of Chemical Engineering, 2019, 27(7): 1485-1497.
[7]	Yixin Ma, Peizhe Cui, Yongkun Wang, Zhaoyou Zhu, Yinglong Wang, Jun Gao. A review of extractive distillation from an azeotropic phenomenon for dynamic control [J]. Chinese Journal of Chemical Engineering, 2019, 27(7): 1510-1522.
[8]	Li Xia, Yuanli Feng, Xiaoyan Sun, Shuguang Xiang. Design of heat exchanger network based on entransy theory [J]. Chin.J.Chem.Eng., 2018, 26(8): 1692-1699.
[9]	Lixia Kang, Yongzhong Liu. A systematic strategy for multi-period heat exchanger network retrofit under multiple practical restrictions [J]. , 2017, 25(8): 1043-1051.
[10]	Jianqiang Deng, Zheng Cao, Dongbo Zhang, Xiao Feng. Integration of energy recovery network including recycling residual pressure energy with pinch technology [J]. , 2017, 25(4): 453-462.
[11]	Yangcheng Lu, Tianyang Li, Rui Wang, Guangsheng Luo. Synthesis of epichlorohydrin from 1,3-dichloropropanol using solid base [J]. , 2017, 25(3): 301-305.
[12]	Yang Yuan, Liang Zhang, Haisheng Chen, Shaofeng Wang, Kejin Huang, Huan Shao. Interpreting the dynamic effect of internal heat integration on reactive distillation columns [J]. , 2017, 25(1): 89-102.
[13]	Lixia Kang, Yongzhong Liu. Minimizing investment cost for multi-period heat exchanger network retrofit by matching heat transfer areas with different strategies [J]. Chin.J.Chem.Eng., 2015, 23(7): 1153-1160.
[14]	Xianli Shi, Li Deng, Fangfang Sun, Jieyu Liang, Xu Deng. Key factors governing alkaline pretreatment of waste activated sludge [J]. Chin.J.Chem.Eng., 2015, 23(5): 842-846.
[15]	Lili Tao, ZhihuaHu, Feng Qian. Design and control of a p-xylene oxidation process [J]. Chin.J.Chem.Eng., 2015, 23(12): 1935-1944.