Fast Learning in Spiking Neural Networks by Learning Rate Adaptation*

Chin.J.Chem.Eng. ›› 2012, Vol. 20 ›› Issue (6): 1219-1224.

• RESEARCH NOTES • Previous Articles

Fast Learning in Spiking Neural Networks by Learning Rate Adaptation^*

FANG Huijuan, LUO Jiliang, WANG Fei

College of Information Science and Engineering, Huaqiao University, Xiamen 361021, China

Received:2012-06-10 Revised:2012-07-31 Online:2012-12-28 Published:2012-12-28
Supported by:
Supported by the National Natural Science Foundation of China (60904018;61203040);the Natural Science Foundation of Fujian Province of China (2009J05147;2011J01352);the Foundation for Distinguished Young Scholars of Higher Education of Fujian Province of China (JA10004);the Science Research Foundation of Huaqiao University (09BS617)

Fast Learning in Spiking Neural Networks by Learning Rate Adaptation^*

方慧娟, 罗继亮, 王飞

College of Information Science and Engineering, Huaqiao University, Xiamen 361021, China

通讯作者: FANG Huijuan,E-mail:huijuan.fang@163.com
基金资助:
Supported by the National Natural Science Foundation of China (60904018;61203040);the Natural Science Foundation of Fujian Province of China (2009J05147;2011J01352);the Foundation for Distinguished Young Scholars of Higher Education of Fujian Province of China (JA10004);the Science Research Foundation of Huaqiao University (09BS617)

Abstract

Abstract: For accelerating the supervised learning by the SpikeProp algorithm with the temporal coding paradigm in spiking neural networks (SNNs),three learning rate adaptation methods (heuristic rule,delta-delta rule,and delta-bar-delta rule),which are used to speed up training in artificial neural networks,are used to develop the training algorithms for feedforward SNN.The performance of these algorithms is investigated by four experiments: classical XOR (exclusive or) problem,Iris dataset,fault diagnosis in the Tennessee Eastman process,and Poisson trains of discrete spikes.The results demonstrate that all the three learning rate adaptation methods are able to speed up convergence of SNN compared with the original SpikeProp algorithm.Furthermore,if the adaptive learning rate is used in combination with the momentum term,the two modifications will balance each other in a beneficial way to accomplish rapid and steady convergence.In the three learning rate adaptation methods,delta-bar-delta rule performs the best.The delta-bar-delta method with momentum has the fastest convergence rate,the greatest stability of training process,and the maximum accuracy of network learning.The proposed algorithms in this paper are simple and efficient,and consequently valuable for practical applications of SNN.

Key words: spiking neural networks, learning algorithm, learning rate adaptation, Tennessee Eastman process

摘要： For accelerating the supervised learning by the SpikeProp algorithm with the temporal coding paradigm in spiking neural networks (SNNs),three learning rate adaptation methods (heuristic rule,delta-delta rule,and delta-bar-delta rule),which are used to speed up training in artificial neural networks,are used to develop the training algorithms for feedforward SNN.The performance of these algorithms is investigated by four experiments: classical XOR (exclusive or) problem,Iris dataset,fault diagnosis in the Tennessee Eastman process,and Poisson trains of discrete spikes.The results demonstrate that all the three learning rate adaptation methods are able to speed up convergence of SNN compared with the original SpikeProp algorithm.Furthermore,if the adaptive learning rate is used in combination with the momentum term,the two modifications will balance each other in a beneficial way to accomplish rapid and steady convergence.In the three learning rate adaptation methods,delta-bar-delta rule performs the best.The delta-bar-delta method with momentum has the fastest convergence rate,the greatest stability of training process,and the maximum accuracy of network learning.The proposed algorithms in this paper are simple and efficient,and consequently valuable for practical applications of SNN.

关键词: spiking neural networks, learning algorithm, learning rate adaptation, Tennessee Eastman process

FANG Huijuan, LUO Jiliang, WANG Fei. Fast Learning in Spiking Neural Networks by Learning Rate Adaptation^*[J]. Chin.J.Chem.Eng., 2012, 20(6): 1219-1224.

方慧娟, 罗继亮, 王飞. Fast Learning in Spiking Neural Networks by Learning Rate Adaptation^*[J]. Chinese Journal of Chemical Engineering, 2012, 20(6): 1219-1224.

References

1 Lazzús,J.A.,"Prediction of flash point temperature of organic compounds using a hybrid method of group contribution+neural network+particle swarm optimization",Chin.J.Chem.Eng.,18 (5),817-823 (2010).
2 Zou,Z.,Yu,D.,Feng,W.,Yu,L.,Guo,N.,"An intelligent neural networks system for adaptive learning and prediction of a bioreactor benchmark process",Chin.J.Chem.Eng.,16 (1),62-66 (2008).
3 Maass,W.,"Networks of spiking neurons: The third generation of neural network models",Neural Networks,10 (9),1659-1671 (1997).
4 Thorpe,S.,Fize D.,Marlot,C.,"Speed of processing in the human visual system",Nature,381 (6582),520-522 (1996).
5 Fang,H.,Wang,Y.,He,J.,"Spiking neural networks for cortical neuronal spike train decoding",Neural Computation,22 (4),1060-1085 (2010).
6 Maass,W.,"Noisy spiking neurons with temporal coding have more computational power than sigmoidal neurons",In: Advances in Neural Information Processing Systems,MIT Press,Cambridge,USA,9,211-217 (1997).
7 Maass,W.,"Lower bounds for the computational power of networks of spiking neurons",Neural Computation,8 (1),1-40 (1996).
8 Bohte,S.M.,Kok,J.N.,La Poutré,H.,"Error-backpropagation in temporally encoded networks of spiking neurons",Neurocomputing,48,17-37 (2002).
9 Yang,J.,Yang,W.,Wu,W.,"A remark on the error-backpropagation learning algorithm for spiking neural networks",Applied Mathematics Letters,25 (8),1118-1120 (2012).
10 Schrauwen,B.,van Campenhout,J.,"Extending spikeprop",In: Proceedings of the International Joint Conference on Neural Networks,IEEE,Piscataway,USA,471-475 (2004).
11 Xin,J.,Embrechts,M.,"Supervised learning with spiking neural networks",In: Proceedings of International Joint Conference on Neural Networks,IEEE,Piscataway,USA,1772-1777 (2001).
12 McKennoch,S.,Liu,D.,Bushnell,L.G.,"Fast modifications of the SpikeProp algorithm",In: Proceedings of the International Joint Conference on Neural Networks,IEEE,Piscataway,USA,3970-3977 (2006).
13 Ghosh-Dastidar,S.,Adeli,H.,"Improved spiking neural networks for EEG classification and epilepsy and seizure detection",Integr.Comput.-Aid E.,14 (3),187-212 (2007).
14 Delshad,E.,Moallem,P.,Monadjemi,S.A.H.,"Spiking neural network learning algorithms: Using learning rates adaptation of gradient and momentum steps",In: 5th International Symposium on Telecommunications (IST),IEEE,944-949 (2010).
15 Gerstner,W.,Kistler,W.,Spiking Neuron Models,Cambridge University Press,England (2002).
16 Vogl,T.P.,Mangis,J.K.,Rigler,A.K.,Zink,W.T.,Alkon,D.L.,"Accelerating the convergence of the back-propagation method",Biol.Cybern.,59 (4),257-263 (1988).
17 Jacobs,R.A.,"Increased rates of convergence through learning rate adaptation",Neural Networks,1,295-307 (1988).
18 Moore,S.C.,"Back-propagation in spiking neural networks",Master Thesis,University of Bath,UK (2002).
19 Downs,J.J.,Vogel,E.F.,"A plant-wide industrial process control problem",Comput.Chem.Eng.,17 (3),245-255 (1993).
20 "Tennessee Eastman Problem for MATLAB",Control Systems Engineering Laboratory,Arizona State University,1998 [2012-06-08],http://csel.asu.edu/downloads/Software/TEmatlab.zip.
21 Lu,N.,Yu,X.,"Fault diagnosis in TE process based on feature selection via second order mutual information",CIESC Journal,60 (9),2252-2258 (2009).
22 Heeger,D.,"Poisson model of spike generation",New York University,2000 [2012-06-08],http://www.cns.nyu.edu/~david/ftp/handouts/poisson.pdf.

[1]	Huihui Gao, Yuan Xu, Xiangbai Gu, Xiaoyong Lin, Qunxiong Zhu. Systematic rationalization approach for multivariate correlated alarms based on interpretive structural modeling and Likert scale [J]. Chin.J.Chem.Eng., 2015, 23(12): 1987-1996.
[2]	Shijin Ren, Zhihuan Song, Maoyun Yang, Jianguo Ren . A novel multimode processmonitoring method integrating LCGMMwith modified LFDA [J]. Chin.J.Chem.Eng., 2015, 23(12): 1970-1980.
[3]	Qunxiong Zhu, Yiwen Jia, Di Peng, Yuan Xu. Study and Application of Fault Prediction Methods with Improved Reservoir Neural Networks [J]. , 2014, 22(7): 812-819.
[4]	JIANG Qingchao, YAN Xuefeng . Statistical Monitoring of Chemical Processes Based on Sensitive Kernel Principal Components [J]. Chin.J.Chem.Eng., 2013, 21(6): 633-643.
[5]	CHEN Xinyi, YAN Xuefeng . Fault Diagnosis in Chemical Process Based on Self-organizing Map Integrated with Fisher Discriminant Analysis [J]. Chin.J.Chem.Eng., 2013, 21(4): 382-387.
[6]	PENG Di, XU Yuan, ZHU Qunxiong . Study and Application of Case-based Extension Fault Diagnosis for Chemical Process [J]. Chin.J.Chem.Eng., 2013, 21(4): 366-375.
[7]	XIE Xiang, SHI Hongbo. Multimode Process Monitoring Based on Fuzzy C-means in Locality Preserving Projection Subspace^* [J]. Chin.J.Chem.Eng., 2012, 20(6): 1174-1179.
[8]	YE Lubin, SHI Xiangrong, LIANG Jun. A Multi-level Approach for Complex Fault Isolation Based on Structured Residuals [J]. , 2011, 19(3): 462-472.
[9]	SONG Kai, WANG Haiqing, LI Ping, FENG Zhigang. Quality Based Prioritized Sensor Fault Monitoring Methodology [J]. , 2008, 16(4): 584-589.

Fast Learning in Spiking Neural Networks by Learning Rate Adaptation^*

Fast Learning in Spiking Neural Networks by Learning Rate Adaptation^*

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 9

Recommended Articles

Metrics

Comments