Fault tolerance in feedforward neural networks using minimax optimization

Abstract

Neural computing is currently being proposed as a viable solution to several problems. The approach of neural computing is to capture the guiding principles that underlie the functioning of the human brain and apply them to computer systems. Neural networks have therefore evolved as parallel systems in which many simple processing elements share the job of working out what is going on, rather than trying to make one fast node do all the work. Among other advantages, this division of labour is claimed to offer fault tolerance. Since many units are involved at one time, the contribution made by a single unit is not too important. So, if one becomes faulty, it is unlikely to significantly affect the others. This distribution of work, known as distributed processing, is supposed to make the network performance tolerant to occasional errors. This ability of the system to function with only some of the processing elements working correctly is known in computing circles as fault tolerance. By virtue of their architecture and processing style, neural networks are believed to be inherently fault tolerant. However, this is not entirely true, since the ability of the network to exhibit fault tolerance depends on the algorithm used to train it. Most connectionist or neural network learning systems use some version of the back-propagation algorithm, which has been shown to produce networks that are not always fault tolerant. This thesis proposes a technique by which fault tolerance can be embedded into a feedforward network, leading to a more robust network that is tolerant to the loss of a node and its weights. The fault tolerance problem for a feedforward network can be formulated as a constrained minimax optimization problem. Two different methods are used to solve this problem:

Sequential Unconstrained Minimization The minimax optimization problem is converted to a minimization of a sequence of unconstrained and continuously differentiable functions. An efficient gradient minimization technique is then applied to perform the unconstrained minimization at each step. Successive modifications are made to the basic algorithm to improve its performance in terms of time. l∞l^\inftyl∞ Norm Approximation An l∞l^\inftyl∞ norm approximation of the objective function of the original minimax optimization problem is used to convert it into an unconstrained minimization problem. This problem is then solved to obtain the results. Networks obtained using these methods, though not totally fault tolerant, exhibit an acceptable degree of partial fault tolerance.