Plefka's mean-field theory and belief networks

Abstract

Mean-field theory is an interesting alternative to Markov Chain Monte Carlo techniques for probabilistic inference in graphical models. In this thesis, we study a powerful mean-field technique proposed by Plefka, in the context of spin glasses. The power of this theory lies in approximating the partition function to any desired degree of accuracy. We propose two alternate derivations of this theory, one from a variational viewpoint and the other from an information-geometric viewpoint. The variational viewpoint establishes that it is more general than the existing theories. The information-geometric framework provides an elegant alternative to the algebraic derivations. Also, in course of the derivation, it is pointed out that sophisticated refinements like the TAP term or the linear response corrections can be easily incorporated in this theory. The major contribution of this thesis is the application of this method to do approximate inference to a class of binary-valued Belief Networks. This application is not straightforward and needs further approximations. A Taylor series-based approximation method is proposed which leads to a computationally simpler but efficient scheme. As an interesting side-step, we use this scheme to show that the multiple cause mixture model is a crude approximation of a two-layer Noisy-OR network. Another important contribution of this thesis is to generalize this method to handle discrete and continuous variables. As in the binary case, this new theory turns out to be more powerful than the existing variational theories. This theory can be used to compute a second-order term both for the discrete and continuous case, which is analogous to the TAP term.