tailieunhanh - Mạng thần kinh thường xuyên cho dự đoán P9

A Class of Normalised Algorithms for Online Training of Recurrent Neural Networks A normalised version of the real-time recurrent learning (RTRL) algorithm is introduced. This has been achieved via local linearisation of the RTRL around the current point in the state space of the network. Such an algorithm provides an adaptive learning rate normalised by the L2 norm of the gradient vector at the output neuron. The analysis is general and also covers simpler cases of feedforward networks and linear FIR filters. | Recurrent Neural Networks for Prediction Authored by Danilo P. Mandic Jonathon A. Chambers Copyright 2001 John Wiley Sons Ltd ISBNs 0-471-49517-4 Hardback 0-470-84535-X Electronic 9 A Class of Normalised Algorithms for Online Training of Recurrent Neural Networks Perspective A normalised version of the real-time recurrent learning RTRL algorithm is introduced. This has been achieved via local linearisation of the RTRL around the current point in the state space of the network. Such an algorithm provides an adaptive learning rate normalised by the L2 norm of the gradient vector at the output neuron. The analysis is general and also covers simpler cases of feedforward networks and linear FIR filters. Introduction Gradient-descent-based algorithms for training neural networks such as the backpropagation backpropagation through time recurrent backpropagation RBP and real-time recurrent learning RTRL algorithm typically suffer from slow convergence when dealing with statistically nonstationary inputs. In the area of linear adaptive filters similar problems with the LMS algorithm have been addressed by utilising normalised algorithms such as NLMS. We therefore introduce a normalised RTRL-based learning algorithm with the idea to impose similar stabilisation and convergence effects on training of RNNs as normalisation imposes on the LMS algorithm. In the area of linear FIR adaptive filters it is shown Soria-Olivas et al. 1998 that a normalised gradient-descent-based learning algorithm can be derived starting from the Taylor series expansion of the instantaneous output error of an adaptive FIR filter given by e k 1 e k de k w d e k A w k AW k e k 1 e k dwi k Aw k 2 2 dwi k dwj k k j k 150 OVERVIEW From the mathematical description of LMS1 from Chapter 2 we have and de k dwi k x k i 1 i 1 2 . N Awi k p k e k x k i 1 i 1 2 . N. Due to the linearity of the FIR filter the second- and higher-order partial derivatives in vanish. Combining - yields