Univariate imputation method for recovering missing data in wastewater treatment process

doi:10.1016/j.cjche.2022.01.033

中国化学工程学报 ›› 2023, Vol. 53 ›› Issue (1): 201-210.DOI: 10.1016/j.cjche.2022.01.033

• Full Length Article • 上一篇下一篇

Univariate imputation method for recovering missing data in wastewater treatment process

Honggui Han, Meiting Sun, Huayun Han, Xiaolong Wu, Junfei Qiao

af005. Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China;
af010. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, China;
af015. Engineering Research Center of Digital Community, Ministry of Education, Beijing 100124, China

收稿日期:2021-09-13 修回日期:2022-01-05 出版日期:2023-01-28 发布日期:2023-04-08
通讯作者: Honggui Han,E-mail:Rechardhan@sina.com
基金资助:
The authors are thankful to the National Key Research and Development Project (No. 2018YFC1900800-5), the National Natural Science Foundation of China (Nos. 61890930-5, 61903010, 6202100), the Beijing Outstanding Young Scientist Program (No. BJJWZYJH01201910005020) and the Beijing Natural Science Foundation (No. KZ202110005009).

Univariate imputation method for recovering missing data in wastewater treatment process

Honggui Han, Meiting Sun, Huayun Han, Xiaolong Wu, Junfei Qiao

af005. Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China;
af010. Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, China;
af015. Engineering Research Center of Digital Community, Ministry of Education, Beijing 100124, China

Received:2021-09-13 Revised:2022-01-05 Online:2023-01-28 Published:2023-04-08
Contact: Honggui Han,E-mail:Rechardhan@sina.com
Supported by:
The authors are thankful to the National Key Research and Development Project (No. 2018YFC1900800-5), the National Natural Science Foundation of China (Nos. 61890930-5, 61903010, 6202100), the Beijing Outstanding Young Scientist Program (No. BJJWZYJH01201910005020) and the Beijing Natural Science Foundation (No. KZ202110005009).

摘要/Abstract

摘要： High-quality data play a paramount role in monitoring, control, and prediction of wastewater treatment process (WWTP) and can effectively ensure the efficient and stable operation of system. Missing values seriously degrade the accuracy, reliability and completeness of the data quality due to network collapses, connection errors and data transformation failures. In these cases, it is infeasible to recover missing data depending on the correlation with other variables. To tackle this issue, a univariate imputation method (UIM) is proposed for WWTP integrating decomposition method and imputation algorithms. First, the seasonal-trend decomposition based on loess method is utilized to decompose the original time series into the seasonal, trend and remainder components to deal with the nonstationary characteristics of WWTP data. Second, the support vector regression is used to approximate the nonlinearity of the trend and remainder components respectively to provide estimates of its missing values. A self-similarity decomposition is conducted to fill the seasonal component based on its periodic pattern. Third, all the imputed results are merged to obtain the imputation result. Finally, six time series of WWTP are used to evaluate the imputation performance of the proposed UIM by comparing with existing seven methods based on two indicators. The experimental results illustrate that the proposed UIM is effective for WWTP time series under different missing ratios. Therefore, the proposed UIM is a promising method to impute WWTP time series.

关键词: Univariate, Self-similarity, Waste water, Algorithm, Integration

Abstract: High-quality data play a paramount role in monitoring, control, and prediction of wastewater treatment process (WWTP) and can effectively ensure the efficient and stable operation of system. Missing values seriously degrade the accuracy, reliability and completeness of the data quality due to network collapses, connection errors and data transformation failures. In these cases, it is infeasible to recover missing data depending on the correlation with other variables. To tackle this issue, a univariate imputation method (UIM) is proposed for WWTP integrating decomposition method and imputation algorithms. First, the seasonal-trend decomposition based on loess method is utilized to decompose the original time series into the seasonal, trend and remainder components to deal with the nonstationary characteristics of WWTP data. Second, the support vector regression is used to approximate the nonlinearity of the trend and remainder components respectively to provide estimates of its missing values. A self-similarity decomposition is conducted to fill the seasonal component based on its periodic pattern. Third, all the imputed results are merged to obtain the imputation result. Finally, six time series of WWTP are used to evaluate the imputation performance of the proposed UIM by comparing with existing seven methods based on two indicators. The experimental results illustrate that the proposed UIM is effective for WWTP time series under different missing ratios. Therefore, the proposed UIM is a promising method to impute WWTP time series.

Key words: Univariate, Self-similarity, Waste water, Algorithm, Integration

Honggui Han, Meiting Sun, Huayun Han, Xiaolong Wu, Junfei Qiao. Univariate imputation method for recovering missing data in wastewater treatment process[J]. 中国化学工程学报, 2023, 53(1): 201-210.

Honggui Han, Meiting Sun, Huayun Han, Xiaolong Wu, Junfei Qiao. Univariate imputation method for recovering missing data in wastewater treatment process[J]. Chinese Journal of Chemical Engineering, 2023, 53(1): 201-210.

参考文献

[1] M.A. Shannon, P.W. Bohn, M. Elimelech, J.G. Georgiadis, B.J. Mariñas, A.M. Mayes, Science and technology for water purification in the coming decades, Nature 452 (7185) (2008) 301–310.https://pubmed.ncbi.nlm.nih.gov/18354474
[2] H.G. Han, X.L. Wu, L.M. Ge, J.F. Qiao, A sludge volume index (SVI) model based on the multivariate local quadratic polynomial regression method, Chin. J. Chem. Eng. 26 (5) (2018) 1071–1077. 10.1016/j.cjche.2017.08.007
[3] W. Wei, P.F. Xia, Z.W. Liu, M. Zuo, A modified active disturbance rejection control for a wastewater treatment process, Chin. J. Chem. Eng. 28 (10) (2020) 2607–2619. 10.1016/j.cjche.2020.06.032
[4] H.G. Han, S.G. Zhu, J.F. Qiao, M. Guo, Data-driven intelligent monitoring system for key variables in wastewater treatment process, Chin. J. Chem. Eng. 26 (10) (2018) 2093–2101. 10.1016/j.cjche.2018.03.027
[5] I. Lizarralde, T. Fernández-Arévalo, A. Manas, E. Ayesa, P. Grau, Model-based opti mization of phosphorus management strategies in Sur WWTP, Madrid, Water Res. 153 (2019) 39–52. 10.1016/j.watres.2018.12.056
[6] H.G. Han, Z. Liu, Y. Hou, J.F. Qiao, Data-driven multiobjective predictive control for wastewater treatment process, IEEE Trans. Ind. Inform. 16 (4) (2020) 2767–2775. 10.1109/TII.2019.2940663
[7] W.D. Tian, Z.J. Liu, L.N. Li, S.F. Zhang, C.K. Li, Identification of abnormal conditions in high-dimensional chemical process based on feature selection and deep learning, Chin. J. Chem. Eng. 28 (7) (2020) 1875–1883. 10.1016/j.cjche.2020.05.003
[8] P. Baraldi, F.D. Maio, D. Genini, E. Zio, Reconstruction of missing data in multidimensional time series by fuzzy similarity, Appl. Soft Comput. 26 (2015) 1–9. 10.1016/j.asoc.2014.09.038
[9] L. Rieger, I. Takács, K. Villez, H. Siegrist, P. Lessard, P.A. Vanrolleghem, Y. Comeau, Data reconciliation for wastewater treatment plant simulation studies-planning for high-quality data and typical sources of errors, Water Environ. Res. 82 (5) (2010) 426–433. 10.2175/106143009x12529484815511
[10] G. Olsson, H. Aspegren, M.K. Nielsen, Operation and control of wastewater treatment—A Scandinavian perspective over 20 years, Water Sci. Technol. 37 (12) (1998) 1–13. 10.1016/S0273-1223(98)00364-3
[11] H.R. Haimi, M. Mulas, F. Corona, R. Vahala, Data-derived soft-sensors for biological wastewater treatment plants: An overview, Environ. Model. Softw. 47 (2013) 88–107. 10.1016/j.envsoft.2013.05.009
[12] N.M. Noor, M.M. Al Bakri Abdullah, A.S. Yahaya, N.A. Ramli, Comparison of linear interpolation method and mean method to replace the missing values in environmental data set, Mater. Sci. Forum 803 (2014) 278–281. 10.4028/www.scientific.net/msf.803.278
[13] G. Hawthorne, P. Elliott, Imputing cross-sectional missing data: Comparison of common techniques, Aust. N. Z. J. Psychiatry 39 (7) (2005) 583–590.https://pubmed.ncbi.nlm.nih.gov/15996139/
[14] Z.W. Wang, L. Wang, Y.Y. Tan, J.F. Yuan, Fault detection based on Bayesian network and missing data imputation for building energy systems, Appl. Therm. Eng. 182 (2021) 116051. 10.1016/j.applthermaleng.2020.116051
[15] Y. Deng, C. Chang, M.S. Ido, Q. Long, Multiple imputation for general missing data patterns in the presence of high-dimensional data“>, Sci. Reports”> 6“> (2016) 21689.https://www.nature.com/articles/srep21689%22%3e
[16] R. Rustum, A.J. Adeloye, Replacing outliers and missing values from activated sludge data using kohonen self-organizing map, J. Environ. Eng. 133 (9) (2007) 909–916. 10.1061/(asce)0733-9372(2007)133:9(909)
[17] H. Tabari, P. Hosseinzadeh Talaee, Reconstruction of river water quality missing data using artificial neural networks, Water Qual. Res. J. 50 (4) (2015) 326–335. ://doi.org/10.2166/wqrjc.2015.044
[18] da Xu, J.Q. Sheng, P.J.H. Hu, T.S. Huang, C.C. Hsu, A deep learning–based unsupervised method to impute missing values in patient records for improved management of cardiovascular patients, IEEE J. Biomed. Heal. Inform. 25 (6) (2021) 2260–2272. 10.1109/JBHI.2020.3033323
[19] Q. Shang, Z.S. Yang, S. Gao, D.R. Tan, An imputation method for missing traffic data based on FCM optimized by PSO-SVR, J. Adv. Transp. 2018 (2018) 2935248. 10.1155/2018/2935248
[20] D. Li, Y.X. Zhou, G.Q. Hu, C.J. Spanos, Handling incomplete sensor measurements in fault detection and diagnosis for building HVAC systems, IEEE Trans. Autom. Sci. Eng. 17 (2) (2020) 833–846. 10.1109/TASE.2019.2948101
[21] A.M. Sefidian, N. Daneshpour, Missing value imputation using a novel grey based fuzzy c-means, mutual information based feature selection, and regression model, Expert Syst. Appl. 115 (2019) 68–94. 10.1016/j.eswa.2018.07.057
[22] N. Shaadan, N.A.M. Rahim, Imputation analysis for time series air quality ({PM}10) data set: A comparison of several methods, J. Phys. Conf. Ser. 1366 (1) (2019) 012107. 10.1088/1742-6596/1366/1/012107
[23] A. Chaudhry, W. Li, A. Basri, F. Patenaude, A method for improving imputation and prediction accuracy of highly seasonal univariate data with large periods of missingness, Wirel. Commun. Mob. Comput. 2019 (2019) 4039758. 10.1155/2019/4039758
[24] S.J. Hadeed, M.K. O'Rourke, J.L. Burgess, R.B. Harris, R.A. Canales, Imputation methods for addressing missing data in short-term monitoring of air pollutants, Sci. Total. Environ. 730 (2020) 139140. 10.1016/j.scitotenv.2020.139140
[25] S. Moritz, A. Sard?a, T. Bartz-Beielstein, M. Zaefferer, J. Stork, Comparison of different methods for univariate time series imputation in R, ArXiv 15 (2015) 3924-3944.
[26] P.D. Allison. Missing data: Quantitative applications in the social sciences, Br. J. Math. Stat. Psychol. 55(1) (2002) 193–196.
[27] V. Audigier, F. Husson, J. Josse, Multiple imputation for continuous variables using a Bayesian principal component analysis, J. Stat. Comput. Simul. 86 (11) (2016) 2140–2156. 10.1080/00949655.2015.1104683
[28] W.O. Yodah, J.M. Kihoro, K.H.O. Arhiany, H.W. Kibunja, Imputation of incomplete non-stationary seasonal time series data, Math. Theory Model. 3(12) (2013) 142-154.
[29] F. Bashir, H.L. Wei, Handling missing data in multivariate time series using a vector autoregressive model-imputation (VAR-IM) algorithm, Neurocomputing 276 (2018) 23–30. 10.1016/j.neucom.2017.03.097
[30] L. Bigaignon, R. Fieuzal, C. Delon, T. Tallec, Combination of two methodologies, artificial neural network and linear interpolation, to gap-fill daily nitrous oxide flux measurements, Agric. For. Meteorol. 291 (2020) 108037. 10.1016/j.agrformet.2020.108037
[31] L. Greco, M. Cuomo, B-Spline interpolation of Kirchhoff-Love space rods, Comput. Methods Appl. Mech. Eng. 256 (2013) 251–269. 10.1016/j.cma.2012.11.017
[32] N. Bokde, F.M. Álvarez, M.W. Beck, K. Kulat, A novel imputation methodology for time series based on pattern sequence forecasting, Pattern Recognit. Lett. 116 (2018) 88–96.https://pubmed.ncbi.nlm.nih.gov/30416234/
[33] S. Chandrasekaran, S. Moritz, M. Zaefferer, J. Stork, T. Bartz-Beilelstein, Data preprocessing: A new algorithm for univariate imputation designed specifically for industrial needs, in: Proceedings of the Workshop Computational Intelligence, Dortmund, Germany, 2016, 1–12.
[34] T.T.H. Phan, É. Poisson Caillault, A. Lefebvre, A. Bigand, Dynamic time warping-based imputation for univariate time series data, Pattern Recognit. Lett. 139 (2020) 139–147. 10.1016/j.patrec.2017.08.019
[35] S. Chiewchanwattana, C. Lursinsap, C.H. Henry Chu, Imputing incomplete time-series data based on varied-window similarity measure of data sequences, Pattern Recognit. Lett. 28 (9) (2007) 1091–1103. 10.1016/j.patrec.2007.01.008
[36] Y.H. Liu, T. Dillon, W.J. Yu, W. Rahayu, F. Mostafa, Missing value imputation for industrial IoT sensor data with large gaps, IEEE Internet Things J. 7 (8) (2020) 6855–6867. 10.1109/JIOT.2020.2970467
[37] R.B. Cleveland, W.S. Cleveland, STL: A seasonal-trend decomposition procedure based on Loess (with discussion), J. Off. Stat. 6(1) (1990) 3–33.
[38] H.T. He, S.C. Gao, T. Jin, S. Sato, X.Y. Zhang, A seasonal-trend decomposition-based dendritic neuron model for financial time series prediction, Appl. Soft Comput. 108 (2021) 107488. 10.1016/j.asoc.2021.107488
[39] F.F. Li, Z.Y. Wang, X. Zhao, E. Xie, J. Qiu, Decomposition-ANN methods for long-term discharge prediction based on fisher's ordered clustering with MESA, Water Resour. Manag. 33 (9) (2019) 3095–3110. ://dx.doi.org/10.1007/s11269-019-02295-8
[40] T.H. Sun, T.Y. Zhang, Y. Teng, Z. Chen, J.K. Fang, Monthly electricity consumption forecasting method based on X12 and STL decomposition model in an integrated energy system, Math. Probl. Eng. 2019 (2019) 9012543. 10.1155/2019/9012543
[41] S. Polwiang, The time series seasonal patterns of dengue fever and associated weather variables in Bangkok (2003-2017), BMC Infect. Dis. 20 (1) (2020) 208.https://pubmed.ncbi.nlm.nih.gov/32164548/
[42] M. Theodosiou, Forecasting monthly and quarterly time series using STL decomposition, Int. J. Forecast. 27 (4) (2011) 1178–1195. 10.1016/j.ijforecast.2010.11.002
[43] L. Qin, W.D. Li, S.J. Li, Effective passenger flow forecasting using STL and ESN based on two improvement strategies, Neurocomputing 356 (2019) 244–256. 10.1016/j.neucom.2019.04.061
[44] K.B. Newhart, R.W. Holloway, A.S. Hering, T.Y. Cath, Data-driven performance analyses of wastewater treatment plants: A review, Water Res. 157 (2019) 498–513.https://pubmed.ncbi.nlm.nih.gov/30981980/
[45] A.D. Kotzapetros, P.A. Paraskevas, A.S. Stasinakis, Design of a modern automatic control system for the activated sludge process in wastewater treatment, Chin. J. Chem. Eng. 23 (8) (2015) 1340–1349. 10.1016/j.cjche.2014.09.053
[46] H.G. Han, H.J. Zhang, Z. Liu, J.F. Qiao, Data-driven decision-making for wastewater treatment process, Control Eng. Pract. 96 (2020) 104305. 10.1016/j.conengprac.2020.104305
[47] D.B. Rubin, Inference and missing data, Biometrika 63 (3) (1976) 581–592. 10.1093/biomet/63.3.581
[48] P. Wang, C.H. Yang, X.M. Tian, D.X. Huang, Adaptive nonlinear model predictive control using an on-line support vector regression updating strategy, Chin. J. Chem. Eng. 22 (7) (2014) 774–781. 10.1016/j.cjche.2014.05.004
[49] S. Moritz, T. Bartz-Beielstein, imputeTS: Time series missing value imputation in R, R J. 9 (1) (2017) 207. 10.32614/rj-2017-009
[50] N. Alavi, J.S. Warland, A.A. Berg, Filling gaps in evapotranspiration measurements for water budget studies: Evaluation of a Kalman filtering approach, Agric. For. Meteorol. 141 (1) (2006) 57–66.

Univariate imputation method for recovering missing data in wastewater treatment process

Univariate imputation method for recovering missing data in wastewater treatment process

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	Bo Yu, Guang Fu, Xinpei Li, Libo Zhang, Jing Li, Hongtao Qu, Dongbin Wang, Qingfeng Dong, Mengmeng Zhang. Arsenic removal from acidic industrial wastewater by ultrasonic activated phosphorus pentasulfide[J]. 中国化学工程学报, 2023, 60(8): 46-52.
[2]	Yanli Zhang, Zhengkun Hou, Dong Yao, Xiaomin Qiu, Hongru Zhang, Peizhe Cui, Yinglong Wang, Jun Gao, Zhaoyou Zhu, Limei Zhong. Energy, exergy, economic and environmental comprehensive analysis and multi-objective optimization of a sustainable zero liquid discharge integrated process for fixed-bed coal gasification wastewater[J]. 中国化学工程学报, 2023, 58(6): 341-354.
[3]	Linlin Su, Meijun Chen, Li Gong, Hua Yang, Chao Chen, Jun Wu, Ling Luo, Gang Yang, Lulu Long. Boost activation of peroxymonosulfate by iron doped K_2-xMn₈O₁₆: Mechanism and properties[J]. 中国化学工程学报, 2023, 57(5): 88-97.
[4]	Xinhao Li, Qing Ye, Jinlong Li, Lingqiang Yan, Xue Jian, Licheng Xie, Jianyu Zhang. Investigation of energy-efficient heat pump assisted heterogeneous azeotropic distillation for separating of acetonitrile/ethyl acetate/n-hexane mixture[J]. 中国化学工程学报, 2023, 55(3): 20-33.
[5]	Aiqin Gao, Xiang Luo, Huanghuang Chen, Aiqin Hou, Hongjuan Zhang, Kongliang Xie. Design of the reactive dyes containing large planar multi-conjugated systems and their application in non-aqueous dyeing[J]. 中国化学工程学报, 2023, 54(2): 264-271.
[6]	Na Wang, Chong Peng, Zhenmin Cheng, Zhiming Zhou. Molecular reconstruction of vacuum gas oils using a general molecule library through entropy maximization[J]. 中国化学工程学报, 2022, 48(8): 21-29.
[7]	Jiaxin Zhang, Wenjia Luo, Yiyang Dai, Yuman Yao. Cycle temporal algorithm-based multivariate statistical methods for fault diagnosis in chemical processes[J]. 中国化学工程学报, 2022, 47(7): 54-70.
[8]	Hualiang An, Rui Wang, Wenhao Wang, Daolai Sun, Xinqiang Zhao, Yanji Wang. A core–shell Ni/SiO₂@TiO₂ catalyst for highly selective one-step synthesis of 2-propylheptanol from n-pentanal[J]. 中国化学工程学报, 2022, 46(6): 104-112.
[9]	Tong Qin, Zhenhao Xi, Ling Zhao, Weikang Yuan. Monte Carlo simulation of sequential structure control of AN-MA-IA aqueous copolymerization by different operation modes[J]. 中国化学工程学报, 2022, 46(6): 231-242.
[10]	Teng Wang, Zihong Xia, Caixia Chen. Computational study of bubble coalescence/break-up behaviors and bubble size distribution in a 3-D pressurized bubbling gas-solid fluidized bed of Geldart A particles[J]. 中国化学工程学报, 2022, 44(4): 485-496.
[11]	Ran An, Shengxin Chen, Shun Hou, Yuting Zhu, Chunhu Li, Xinbao Zhu, Ruixia Liu, Weizhong An. Simulation and design of a heat-integrated double-effect reactive distillation process for propylene glycol methyl ether production[J]. 中国化学工程学报, 2022, 52(12): 103-114.
[12]	Yuyang Kang, Yiqing Luo, Xigang Yuan. Recent progress on equation-oriented optimization of complex chemical processes[J]. 中国化学工程学报, 2022, 41(1): 162-169.
[13]	Yang Lü, Kai Lu, Youxiang Bai, Yulong Ma, Yongsheng Ren. Fouling characteristics of 90° elbow in high salinity wastewater from coal chemical industry[J]. 中国化学工程学报, 2021, 35(7): 143-151.
[14]	Zhiquan Wang, Liang Wang, Zhihong Yuan, Bingzhen Chen. Data-driven optimal operation of the industrial methanol to olefin process based on relevance vector machine[J]. 中国化学工程学报, 2021, 34(6): 106-115.
[15]	Tongtong Zhang, Xiaohui She, Yulong Ding. A power plant for integrated waste energy recovery from liquid air energy storage and liquefied natural gas[J]. 中国化学工程学报, 2021, 34(6): 242-257.