Minimization Problems Based On A Parametric Family Of Relative Entropies
Abstract
We study minimization problems with respect to a oneparameter family of generalized relative entropies. These relative entropies, which we call relative entropies (denoted I (P; Q)), arise as redundancies under mismatched compression when cumulants of compression lengths are considered instead of expected compression lengths. These parametric relative entropies are a generalization of the usual relative entropy (KullbackLeibler divergence). Just like relative entropy, these relative entropies behave like squared Euclidean distance and satisfy the Pythagorean property. We explore the geometry underlying various statistical models and its relevance to information theory and to robust statistics. The thesis consists of three parts.
In the first part, we study minimization of I (P; Q) as the first argument varies over a convex set E of probability distributions. We show the existence of a unique minimizer when the set E is closed in an appropriate topology. We then study minimization of I on a particular convex set, a linear family, which is one that arises from linear statistical constraints. This minimization problem generalizes the maximum Renyi or Tsallis entropy principle of statistical physics. The structure of the minimizing probability distribution naturally suggests a statistical model of powerlaw probability distributions, which we call an powerlaw family. Such a family is analogous to the exponential family that arises when relative entropy is minimized subject to the same linear statistical constraints.
In the second part, we study minimization of I (P; Q) over the second argument. This minimization is generally on parametric families such as the exponential family or the  powerlaw family, and is of interest in robust statistics ( > 1) and in constrained compression settings ( < 1).
In the third part, we show an orthogonality relationship between the powerlaw family and an associated linear family. As a consequence of this, the minimization of I (P; ), when the second argument comes from an powerlaw family, can be shown to be equivalent to a minimization of I ( ; R), for a suitable R, where the first argument comes from a linear family. The latter turns out to be a simpler problem of minimization of a quasi convex objective function subject to linear constraints. Standard techniques are available to solve such problems, for example, via a sequence of convex feasibility problems, or via a sequence of such problems but on simpler singleconstraint linear families.
Collections
Related items
Showing items related by title, author, creator and subject.

On Generalized Measures Of Information With Maximum And Minimum Entropy Prescriptions
Dukkipati, Ambedkar (20080129)KullbackLeibler relativeentropy or KLentropy of P with respect to R deﬁned as ∫xlnddPRdP , where P and R are probability measures on a measurable space (X, ), plays a basic role in the deﬁnitions of classical information ... 
Higher Spins, Entanglement Entropy And Holography
Datta, Shouvik (20170821)The idea of holography [1, 2] finds a concrete realization in form of the AdS/CFT correspondence [3, 4]. This duality relates a field theory with conformal symmetries to quantum gravity living in one higher dimension. In ... 
Lessons for Gravity from Entanglement
Bhattacharyya, ArpanOne of the recent fundamental developments in theoretical high energy physics is the AdS/CFT correspondence [1, 2, 3, 4] which posits a relationship between Quantum Field Theories (QFT) in a given dimension and String ...