Analytic and numerical studies of neural network models of associative memory

Abstract

In this thesis, we have used analytical and numerical methods to study the behaviour of neural network models for the storage, classification and associative recall of information. In Part A, we have studied a class of hierarchical neural network models for the storage and retreival of strongly correlated memories. This class of models was introduced by Dotsenko. In Section 1 of Part A, we have shown that the models considered by Dotsenko have a serious shortcoming if the patterns stored in each cluster at the lowest level of the hierarchy are, as originally proposed, random binary sequences. The problem is that these models are not able to detect or correct errors in categorization which may be present in the input pattern. In Sections 2 and 3 of Part A we have described two different models vrhich are constructed with a view towards overcoming this lindtation of the original models. Analytic and numerical calculations discussed in detail in these sections show that both the models are able to recognize, and also to correct in a majority of cases, any error in categorization present in the input. The first model, which uses fields conjugate to the patterns stored in the lower level of the hierarchy, is the simpler one. It, however, has the disadvantage of being able to store only a small number of patterns in each of the lower level clusters. The second model does not have this limitation but it involves a more complicated multispin interaction. In Part B, we have studied a neural network model in which individual memories are stored in limit cycles. In this model, there are two distinct time scales, one describing the convergence of the system to the nearest memory state and the other one, determined by the parameter "C , describing the transition between a memorized pattern and its complement. In Section 1 we have shown that the model with cycles performs better than the Hopfield model as a pattern classifier if the memory loading level and the degree of corruption of the input patterns are high. This result was obtained from an analytic calculation on an anisotropically diluted version of the model and a numerical study of the fully connected model. In Section 2 we have described a third model for hierarchical storage of correlated memories using the principles o£ the neural network model for storing limit cycles. Of the two Ci/ni5 scales, one describes the convergence of the system to the closest memory and the other one, determined by X , describes the process of correcting any error in categorization that may be present in a given input. The problems associated with spurious stable states are present, in varying degrees, in all the three iiiodals for hierarchical neural networks. The problem is least severe in the third model. Our numerical simulations of this model indicate that the system always converges to a memory state with thfi correct sign of magnetization if the initial state is close to a memory or its complement. The hierarchical neural network models studied here belong to a more genereil class of neural network models which may be constructed to address the following problem. Consider a network consisting of m clusters each containig n spins. Each cluster

stores p patterns which may or may not be correlated among themselves. By combining the patterns stored in all clusters, we m get (2p) distinct but correlated mn - bit patterns if a pattern and its complement are considered to be distinct. The problem is to determine a scheme for connecting the clusters (either directly or via a second set of spins) in such a way that only a m selected subset of the (2p) patterns correspond to stable memory statefi and the remaining ones are unstable. A network which has this ’pr'operty would be useful in many contexts. For example, the memories stored in each cluster may represent the letters in the alphc,bet, and the overall patterns composed of appropriate memories selected from the clusters may represent meaningful word:'.. In another example, each cluster may store the digits 0-9 and the overall pattern may represent m digit numbers to be memorized. It is hoped that the models studied in this thesis would be useful in this more general context. The neural network models with limit cycles have gan<;rated a lot of interest among neurobiologists as models for Cen’rral Pattern Generator (CPG) . Neurobiological studies have r“VJ;aled (Kristan, 1980) that the CPG’s, which are groups of specialized neurons whose activity is controlled by an internal clock, are responsible for rythmic movements such as respiration, j.oc-orr.otion and the beating of the heart. However it remains to be s.een to what extent neural network model with limit cycles can model such behaviour.