Application of trust region on methods to learning in feed forward neural networks

Abstract

This thesis presents a novel algorithm for training feed-forward neural networks, addressing the limitations of the widely used back-propagation and Quickprop algorithms. While Quickprop is significantly faster than standard back-propagation, its performance heavily depends on problem-specific parameters and often exhibits oscillatory error behavior during training. To overcome these drawbacks, we propose a new algorithm based on restricted step or trust region methods. The approach constructs independent quadratic models of the objective function for each network weight and determines trust regions where these models are valid within a specified tolerance. These models are formed at the output layer and propagated backward, similar to error propagation in back-propagation. Weight updates are performed by minimizing the quadratic models within their trust regions, and the process is iteratively repeated until convergence. Empirical evaluation on benchmark problems shows that the proposed algorithm is at least three times faster than Quickprop and scales efficiently with increasing problem size. Moreover, it eliminates the oscillatory behavior typically observed in Quickprop, offering a more stable and robust training process.