Accelerating parallel tangent learning for neural networks through dynamic self adaptation

Abstract

In the gradient based learning algorithms, the momentum has usually an improving effect in convergence rate and decreasing the zigzagging phenomena but sometimes it causes the convergence rate to decrease. The Parallel tangent (Partan) gradient is used as deflecting method to improve the convergence. In this paper, we modify the gradient Partan algorithm for learning the neural networks by using two different learning rates, one for gradient search and the other for accelerating through parallel tangent, respectively. Moreover, the dynamic self adaptation of learning rate is used to improve the performance. In dynamic self adaptation, each learning rate is adapted locally to the cost function landscape and the previous learning rate. Finally we test the proposed algorithm, called accelerated Partan on various problems such as xor and encoders. We compare the results with those of the dynamic self adaptation of learning rate and momentum.