Proof of Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing

Each result cited is universally quantified over the data in its own statement. Throughout, a real-valued function $u$ on a measurable space is measurable exactly when $\{u>c\}$ belongs to the $\sigma$-algebra for every real $c$, by Measure Spaces and the Lebesgue Integral: Standing Notation §measurable; for a nonnegative integrable function the integral of Integrable Function and the Lebesgue Integral is its integral as a nonnegative measurable function, its negative part being $0$; and we use that $\exp$ is positive and strictly increasing with $\exp(0)=1$ and $\exp(u+v)=\exp(u)\exp(v)$, by claims 1, 2 and 4 of Basic Properties of the Exponential Function, and that $\log$ is its inverse with $\log(st)=\log s+\log t$, by The Natural Logarithm, so that $\log$ is strictly increasing, $\log1=0$ and $\exp(u-\log Z)=\exp(u)/Z$ for real $u$ and positive $Z$. The indicator integral $\int\mathbf{1}_{A}\,d\mu=\mu(A)$ is The Integral of an Indicator Function is the Measure of the Set.

Claim 1. Let $M$ be a bound for $h$ and let $\mu$ be a probability measure on $(S,\mathcal{S})$. The function $|h|$ is measurable by claim 4 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, and $|h|\le M\mathbf{1}_{S}$ pointwise, so by Linearity and Monotonicity of the Lebesgue Integral §nonnegative and the indicator integral $\int_{S}|h|\,d\mu\le M\mu(S)=M<\infty$; hence $h$ is integrable with respect to $\mu$ by the criterion of Integrable Function and the Lebesgue Integral. For real $c\le0$ the set $\{\exp\circ h>c\}$ is $S$, since $\exp$ is positive; for $c>0$ it equals $\{h>\log c\}\in\mathcal{S}$, since $\exp$ is strictly increasing and $\exp(\log c)=c$. So $\exp\circ h$ is measurable, and $\exp(-M)\le\exp(h(s))\le\exp(M)$ for every $s\in S$; thus $\exp\circ h$ is bounded measurable, hence $\gamma$-integrable by what was just shown, and Linearity and Monotonicity of the Lebesgue Integral §integrable, applied to $\exp(-M)\mathbf{1}_{S}\le\exp\circ h$, gives $\int_{S}\exp\circ h\,d\gamma\ge\exp(-M)\gamma(S)=\exp(-M)>0$.

Claim 2. By the definition of finite relative entropy there is a density $f$ of $\nu$ with respect to $\gamma$, a measurable $f:S\to\mathbb{R}$ with $f\ge0$, such that $\phi\circ f$ is $\gamma$-integrable and $H(\nu\,|\,\gamma)=\int_{S}\phi\circ f\,d\gamma$. By The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities §uniqueness, applied with $h_{1}=h_{2}=f$, $f$ is $\gamma$-integrable with $\int_{S}f\,d\gamma=\nu(S)=1$. Since $\nu$ is the measure with density $f$ with respect to $\gamma$, claim 3 of Image Measures, Measures with Densities, and Change of Variables gives: $(\star)$ for every measurable $u:S\to\mathbb{R}$ integrable with respect to $\nu$, the function $uf$ is $\gamma$-integrable and $\int_{S}u\,d\nu=\int_{S}uf\,d\gamma$. The derivation of $(\star)$ and of the integrability of $f$ used only that $f$ is a density of $\nu$ with respect to $\gamma$. Now let $h$ be bounded measurable and $Z=\int_{S}\exp\circ h\,d\gamma$, a positive real by Claim 1. For $s\in S$, The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §young with $a=h(s)-\log Z+1$ and the number $f(s)$ in place of its $s$ gives

$$h(s)f(s)+(1-\log Z)f(s)\le\phi(f(s))+\frac{\exp(h(s))}{Z}.$$

The left side is $\gamma$-integrable by $(\star)$ for $u=h$ (which is $\nu$-integrable by Claim 1) and the integrability of $f$; the right side is $\gamma$-integrable by hypothesis and Claim 1. Integrating, by Linearity and Monotonicity of the Lebesgue Integral §integrable and $(\star)$, gives $\int_{S}h\,d\nu+1-\log Z\le H(\nu\,|\,\gamma)+1$, that is $\Lambda^{\gamma}_{h}(\nu)\le H(\nu\,|\,\gamma)$. For $h=0$, which is bounded measurable by claim 1 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, $\int_{S}h\,d\nu=0$ and $\int_{S}\exp\circ h\,d\gamma=\gamma(S)=1$, so $\Lambda^{\gamma}_{0}(\nu)=0$ and $0\le H(\nu\,|\,\gamma)$.

Claim 3. First let $A\in\mathcal{S}$ with $\gamma(A)=0$, and let $t$ be a positive real. The function $t\mathbf{1}_{A}$ is measurable by claims 1 and 2 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions and has bound $t$, so it is bounded measurable, and $\int_{S}t\mathbf{1}_{A}\,d\nu=t\nu(A)$ by Claim 1, Linearity and Monotonicity of the Lebesgue Integral §integrable and the indicator integral. Since $\exp(0)=1$, pointwise $\exp\circ(t\mathbf{1}_{A})=\mathbf{1}_{S}+(\exp(t)-1)\mathbf{1}_{A}$, so for every probability measure $\mu$ on $(S,\mathcal{S})$, by the same references,

$$(\dagger)\qquad\int_{S}\exp\circ(t\mathbf{1}_{A})\,d\mu=1+(\exp(t)-1)\mu(A).$$

With $\mu=\gamma$ this is $1$, so $t\nu(A)=\Lambda^{\gamma}_{t\mathbf{1}_{A}}(\nu)\le C$ for every positive $t$. If $\nu(A)>0$, then $t=(|C|+1)/\nu(A)$ gives $|C|+1\le C$, which is impossible; so $\nu(A)=0$. The measure $\gamma$ is $\sigma$-finite (take $X_{m}=S$ for every $m$ in Measure, Measure Space, and Probability Measure) and $\nu$ is finite, so The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities §existence gives a density $f$ of $\nu$ with respect to $\gamma$; as noted in Claim 2, $f$ is $\gamma$-integrable with $\int_{S}f\,d\gamma=1$ and $(\star)$ holds. For $n\in\mathbb{N}$ let $f_{n}=\min(\max(f,\frac{1}{n+1}),n+1)$, measurable by claims 1 and 4 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, with $\frac{1}{n+1}\le f_{n}\le n+1$, and let $g_{n}=\log\circ f_{n}$. For real $c$, $\{g_{n}>c\}=\{f_{n}>\exp(c)\}\in\mathcal{S}$, and $|g_{n}|\le\log(n+1)$ because $\log$ is increasing and $\log\frac{1}{n+1}=-\log(n+1)$; so $g_{n}$ is bounded measurable with $\exp\circ g_{n}=f_{n}$. By hypothesis, $(\star)$ for $u=g_{n}$, and the inequality $\log x\le x-1$ of The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §log,

$$(1)\qquad\int_{S}fg_{n}\,d\gamma\le C+\log\int_{S}f_{n}\,d\gamma\le C+\varepsilon_{n},\qquad\varepsilon_{n}=\int_{S}f_{n}\,d\gamma-1 .$$

If $f(s)>0$ then $f_{n}(s)=f(s)$ as soon as $n+1\ge\max(f(s),1/f(s))$, and if $f(s)=0$ then $f_{n}(s)=\frac{1}{n+1}$; so $f_{n}\to f$ pointwise, and $0\le f_{n}\le f+1$ with $f+1$ $\gamma$-integrable. By Dominated Convergence Theorem, $\int_{S}f_{n}\,d\gamma\to\int_{S}f\,d\gamma=1$, so $\varepsilon_{n}\to0$. Let $\psi=\phi\circ f$, measurable by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §continuous, let $Q=\{-f>-1\}=\{f<1\}\in\mathcal{S}$ and $P=S\setminus Q$. For $s\in Q$ we have $f(s)<1\le n+1$, so $f_{n}(s)=\max(f(s),\frac{1}{n+1})\in[f(s),1]$; hence if $f(s)>0$ then $\log f(s)\le g_{n}(s)\le0$ and $|f(s)g_{n}(s)|\le-\phi(f(s))\le\exp(-1)$ by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §lower, while $f(s)g_{n}(s)=0=\psi(s)$ if $f(s)=0$; and $f(s)g_{n}(s)=\psi(s)$ once $n+1\ge1/f(s)$ when $f(s)>0$. The functions $\mathbf{1}_{Q}fg_{n}$ are measurable by claims 1 and 3 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, converge pointwise to $\mathbf{1}_{Q}\psi$ and are bounded in absolute value by the $\gamma$-integrable constant $\exp(-1)$, so by Dominated Convergence Theorem the function $\mathbf{1}_{Q}\psi$ is $\gamma$-integrable and $q_{n}=\int_{S}\mathbf{1}_{Q}fg_{n}\,d\gamma$ converges to $q=\int_{S}\mathbf{1}_{Q}\psi\,d\gamma$. For $s\in P$ we have $f(s)\ge1\ge\frac{1}{n+1}$, so $f_{n}(s)=\min(f(s),n+1)\ge1$; hence $u_{n}=\mathbf{1}_{P}fg_{n}$ satisfies $0\le u_{n}\le u_{n+1}$ everywhere, and $u_{n}(s)=\mathbf{1}_{P}(s)\psi(s)$ once $n+1\ge f(s)$, so $\mathbf{1}_{P}\psi$ is the pointwise least upper bound of $(u_{n})$ and Monotone Convergence Theorem gives $\int_{S}\mathbf{1}_{P}\psi\,d\gamma=\sup_{n}\int_{S}u_{n}\,d\gamma$ in $[0,\infty]$. Since $fg_{n}$ is $\gamma$-integrable by $(\star)$, so is $u_{n}=fg_{n}-\mathbf{1}_{Q}fg_{n}$, and by (1) $\int_{S}u_{n}\,d\gamma\le C+\varepsilon_{n}-q_{n}$. For $n\le k$ this gives $\int_{S}u_{n}\,d\gamma\le\int_{S}u_{k}\,d\gamma\le C+\varepsilon_{k}-q_{k}$, and letting $k\to\infty$ yields $\int_{S}u_{n}\,d\gamma\le C-q$. Hence $\int_{S}\mathbf{1}_{P}\psi\,d\gamma\le C-q<\infty$, so the nonnegative function $\mathbf{1}_{P}\psi$ is $\gamma$-integrable, and by Linearity and Monotonicity of the Lebesgue Integral §integrable so is $\psi=\mathbf{1}_{P}\psi+\mathbf{1}_{Q}\psi$, with $\int_{S}\psi\,d\gamma\le(C-q)+q=C$. By Relative Entropy of Probability Measures §relative-entropy, $\nu$ has finite relative entropy with respect to $\gamma$ and $H(\nu\,|\,\gamma)=\int_{S}\psi\,d\gamma\le C$.

Claim 4. The set $L$ of the numbers $\Lambda^{\gamma}_{h}(\nu)$, $h$ bounded measurable, contains $\Lambda^{\gamma}_{0}(\nu)$ and is bounded above by $H(\nu\,|\,\gamma)$ by Claim 2; let $\lambda$ be its least upper bound, so $\lambda\le H(\nu\,|\,\gamma)$. Since $\Lambda^{\gamma}_{h}(\nu)\le\lambda$ for every bounded measurable $h$, Claim 3 with $C=\lambda$ gives $H(\nu\,|\,\gamma)\le\lambda$. Hence $H(\nu\,|\,\gamma)=\lambda$.

Claim 5. Write $Z_{g}=\int_{S}\exp\circ(tg)\,d\gamma$. Since $tg\ge0$ and $\exp$ is increasing, $\mathbf{1}_{S}\le\exp\circ(tg)$, so $Z_{g}\ge\gamma(S)=1$ by Linearity and Monotonicity of the Lebesgue Integral §integrable. For $m\in\mathbb{N}$ let $h_{m}=t\min(g,m)$, measurable by claims 1, 2 and 4 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, with $0\le h_{m}\le tm$, hence bounded measurable. As $h_{m}\le tg$ and $\exp$ is increasing, $\exp\circ h_{m}\le\exp\circ(tg)$, so $\int_{S}\exp\circ h_{m}\,d\gamma\le Z_{g}$ by the same claim, and since $\log$ is increasing Claim 2 gives

$$t\int_{S}\min(g,m)\,d\nu=\Lambda^{\gamma}_{h_{m}}(\nu)+\log\int_{S}\exp\circ h_{m}\,d\gamma\le H(\nu\,|\,\gamma)+\log Z_{g}.$$

The functions $\min(g,m)$ are nonnegative, measurable and nondecreasing in $m$, and equal $g(s)$ at $s$ once $m\ge g(s)$, so by Monotone Convergence Theorem $\int_{S}g\,d\nu=\sup_{m}\int_{S}\min(g,m)\,d\nu\le\frac{1}{t}(H(\nu\,|\,\gamma)+\log Z_{g})<\infty$. Hence $g$ is $\nu$-integrable by Integrable Function and the Lebesgue Integral, and the displayed inequality of the statement holds.

Claim 6. Let $t$ be a positive real. As in Claim 3, $t\mathbf{1}_{A}$ is bounded measurable with $\int_{S}t\mathbf{1}_{A}\,d\nu=t\nu(A)$, and $(\dagger)$ holds for $\mu=\gamma$. If $\gamma(A)=0$, then $(\dagger)$ equals $1$ and Claim 2 gives $t\nu(A)\le H(\nu\,|\,\gamma)$ for every positive $t$; if $\nu(A)>0$, then $t=(|H(\nu\,|\,\gamma)|+1)/\nu(A)$ gives a contradiction, so $\nu(A)=0$. If $\gamma(A)>0$, let $t=\log(1+1/\gamma(A))$, positive since $1+1/\gamma(A)>1$; then $\exp(t)-1=1/\gamma(A)$, so $(\dagger)$ equals $2$, and Claim 2 gives $t\nu(A)-\log2\le H(\nu\,|\,\gamma)$, which is the asserted inequality.

Claim 7. By claim 1 of Image Measures, Measures with Densities, and Change of Variables, $T_{\#}\nu$ and $T_{\#}\gamma$ are probability measures on $(S',\mathcal{S}')$. Let $h:S'\to\mathbb{R}$ be bounded measurable with respect to $\mathcal{S}'$. For every Borel set $B$ of the real line, $(h\circ T)^{-1}(B)=T^{-1}(h^{-1}(B))\in\mathcal{S}$, as $h^{-1}(B)\in\mathcal{S}'$; so $h\circ T$ is measurable in the sense of Measurable Function and Real-Valued Measurable Function, and it has the bounds of $h$, so it is bounded measurable on $(S,\mathcal{S})$. Claims 1 to 3 have been proved for an arbitrary measurable space and arbitrary probability measures on it, so they apply to $(S',\mathcal{S}')$ with $T_{\#}\nu$ and $T_{\#}\gamma$. By Claim 1 for the measurable space $(S',\mathcal{S}')$, $h$ is $T_{\#}\nu$-integrable and $\exp\circ h$ is bounded measurable, hence $T_{\#}\gamma$-integrable; so claim 2 of Image Measures, Measures with Densities, and Change of Variables, applied once with $\nu$ and once with $\gamma$ as the measure on $(S,\mathcal{S})$, gives $\int_{S'}h\,d(T_{\#}\nu)=\int_{S}h\circ T\,d\nu$ and $\int_{S'}\exp\circ h\,d(T_{\#}\gamma)=\int_{S}\exp\circ(h\circ T)\,d\gamma$. Hence, by Claim 2 applied to $h\circ T$,

$$\Lambda^{T_{\#}\gamma}_{h}(T_{\#}\nu)=\Lambda^{\gamma}_{h\circ T}(\nu)\le H(\nu\,|\,\gamma).$$

This holds for every bounded measurable $h$ on $(S',\mathcal{S}')$, so Claim 3, applied to the measurable space $(S',\mathcal{S}')$, the probability measures $T_{\#}\nu$ and $T_{\#}\gamma$ and $C=H(\nu\,|\,\gamma)$, shows that $T_{\#}\nu$ has finite relative entropy with respect to $T_{\#}\gamma$ and $H(T_{\#}\nu\,|\,T_{\#}\gamma)\le H(\nu\,|\,\gamma)$.