Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing

In the settings of The Real Numbers: Standing Notation and Background and Measure Spaces and the Lebesgue Integral: Standing Notation, the measure space of the latter being instantiated at each use by the measure space named, let $(S,\mathcal{S})$ be a measurable space and let $\nu$ and $\gamma$ be probability measures on it. Finite relative entropy with respect to $\gamma$ and $H(\nu\,|\,\gamma)$ are those of that definition; $\exp$ is the exponential function and $\log$ the natural logarithm. A function $h:S\to\mathbb{R}$ is called bounded measurable if it is measurable with respect to $\mathcal{S}$ and bounded.¶ For bounded measurable $h$, claim 1 below makes $h$ integrable with respect to $\nu$ and $\int_{S}\exp\circ h\,d\gamma$ a positive real number, and we write¶

$$\Lambda^{\gamma}_{h}(\nu)=\int_{S}h\,d\nu-\log\int_{S}\exp\circ h\,d\gamma .$$

1. (The functional)¶ Let $h:S\to\mathbb{R}$ be bounded measurable. Then $h$ is integrable with respect to every probability measure on $(S,\mathcal{S})$, the function $\exp\circ h$ is bounded measurable, and $\int_{S}\exp\circ h\,d\gamma$ is a positive real number.

2. (Gibbs inequality)¶ If $\nu$ has finite relative entropy with respect to $\gamma$, then $\Lambda^{\gamma}_{h}(\nu)\le H(\nu\,|\,\gamma)$ for every bounded measurable $h:S\to\mathbb{R}$; in particular $0\le H(\nu\,|\,\gamma)$.

3. (Variational criterion)¶ Let $C\in\mathbb{R}$ be such that $\Lambda^{\gamma}_{h}(\nu)\le C$ for every bounded measurable $h:S\to\mathbb{R}$. Then $\nu$ has finite relative entropy with respect to $\gamma$, and $H(\nu\,|\,\gamma)\le C$.

4. (Variational formula)¶ If $\nu$ has finite relative entropy with respect to $\gamma$, then $H(\nu\,|\,\gamma)$ is the least upper bound of the set of real numbers $\Lambda^{\gamma}_{h}(\nu)$, $h:S\to\mathbb{R}$ bounded measurable.

5. (Entropy inequality)¶ Let $\nu$ have finite relative entropy with respect to $\gamma$, let $g:S\to\mathbb{R}$ be measurable with respect to $\mathcal{S}$ with $0\le g(s)$ for every $s\in S$, and let $t$ be a positive real number such that $\exp\circ(tg)$ is integrable with respect to $\gamma$. Then $1\le\int_{S}\exp\circ(tg)\,d\gamma$, the function $g$ is integrable with respect to $\nu$, and

$$\int_{S}g\,d\nu\le\frac{1}{t}\Bigl(H(\nu\,|\,\gamma)+\log\int_{S}\exp\circ(tg)\,d\gamma\Bigr).$$

6. (Small sets)¶ Let $\nu$ have finite relative entropy with respect to $\gamma$ and let $A\in\mathcal{S}$. If $\gamma(A)=0$, then $\nu(A)=0$. If $\gamma(A)>0$, then

$$\nu(A)\,\log\Bigl(1+\frac{1}{\gamma(A)}\Bigr)\le H(\nu\,|\,\gamma)+\log2 .$$

7. (Data processing)¶ Let $(S',\mathcal{S}')$ be a measurable space, let $T:S\to S'$ be measurable with respect to $\mathcal{S}$ and $\mathcal{S}'$, and let $T_{\#}\nu$ and $T_{\#}\gamma$ be the image measures $B\mapsto\nu(T^{-1}(B))$ and $B\mapsto\gamma(T^{-1}(B))$ of claim 1 of that lemma, probability measures on $(S',\mathcal{S}')$ by that claim. If $\nu$ has finite relative entropy with respect to $\gamma$, then $T_{\#}\nu$ has finite relative entropy with respect to $T_{\#}\gamma$, and $H(T_{\#}\nu\,|\,T_{\#}\gamma)\le H(\nu\,|\,\gamma)$.