Proof of Relative Entropy on Euclidean Space: the Gibbs Inequality, a Variational Criterion for Finite Relative Entropy, and Closed Sublevel Sets under Weak Convergence

Each result cited below is universally quantified over the data in its own statement.

Preliminaries. For a bounded Borel $h:\mathbb{R}^m\to\mathbb{R}$ we fix a bound $b\ge0$, so that $-b\le h(x)\le b$ for every $x$ by claim 6 of Properties of the Absolute Value in an Ordered Field; every bounded Borel map is integrable with respect to every member of $\mathcal{P}(\mathbb{R}^m)$ by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §measures. For a nonnegative integrable function, its integral in the sense of the integrable case equals its integral as a $[0,\infty]$-valued map, its negative part being $0$; this is used without comment. We also use three elementary facts about $\exp$ and $\log$.

(E1) For $t>0$ and real $a$: $a<\log t$ if and only if $\exp(a)<t$, and $a\le\log t$ if and only if $\exp(a)\le t$. This follows from $\exp(\log t)=t$ and $\log(\exp(a))=a$ (The Natural Logarithm) and the strict monotonicity of $\exp$ (claim 4 of Basic Properties of the Exponential Function). Also $\log1=0$, as $\exp(0)=1$ by claim 1 there.

(E2) If $u,v\in[-b,b]$ then $|\exp(u)-\exp(v)|\le\exp(b)|u-v|$. Indeed, let $u\le v$. By The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §exp, $\exp(u-v)\ge1+u-v$; multiplying by $\exp(v)>0$ (claim 2 of Basic Properties of the Exponential Function, claim 5 of Elementary Arithmetic in an Ordered Field) and using claim 1 of Basic Properties of the Exponential Function gives $\exp(u)\ge\exp(v)-\exp(v)(v-u)$. Together with claim 4 there, $0\le\exp(v)-\exp(u)\le\exp(v)(v-u)\le\exp(b)(v-u)$, the last step by $\exp(v)\le\exp(b)$ and claim 5 of Elementary Arithmetic in an Ordered Field. The case $v\le u$ is symmetric.

(E3) If $s,t\ge\exp(-b)$ then $|\log t-\log s|\le\exp(b)|t-s|$. Indeed, let $s\le t$. Then $\log t=\log s+\log(t/s)$ by The Natural Logarithm, and $\log(t/s)\le t/s-1=(t-s)s^{-1}$ by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §log. Multiplying $\exp(-b)\le s$ by the positive number $s^{-1}\exp(b)$ (claims 5 and 7 of Elementary Order Arithmetic in an Ordered Field, claim 5 of Elementary Arithmetic in an Ordered Field) and using $\exp(-b)\exp(b)=1$ gives $s^{-1}\le\exp(b)$. Also $\log s\le\log t$ by (E1), since $\exp(\log s)=s\le t$. Hence $0\le\log t-\log s\le\exp(b)(t-s)$; the case $t\le s$ is symmetric.

Claim 1 (The exponential moment). Let $h$ be bounded Borel with bound $b$. For real $a\le0$ the set $\{x:a<\exp(h(x))\}$ is $\mathbb{R}^m$, by claim 2 of Basic Properties of the Exponential Function; for $a>0$ it is $\{x:\log a<h(x)\}$ by (E1), a Borel set. So $\exp\circ h$ is Borel by claim 3 of Rational Intervals and Rays Generate the Borel Sigma-Algebra of the Real Line. By claim 4 of Basic Properties of the Exponential Function, $0<\exp(-b)\le\exp(h(x))\le\exp(b)$ for every $x$, so $\exp\circ h$ is bounded, hence $\gamma$-integrable by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §measures. By claim 2 of Linearity and Monotonicity of the Lebesgue Integral and The Integral of an Indicator Function is the Measure of the Set,

so $\int\exp\circ h\,d\gamma$ is a positive real number.

Claim 2 (Gibbs inequality). By Relative Entropy of Probability Measures §relative-entropy, $\nu$ has a density $f$ with respect to $\gamma$ such that $\phi\circ f$ is $\gamma$-integrable and $H(\nu\,|\,\gamma)=\int\phi\circ f\,d\gamma$. By The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities §uniqueness (with $h_1=h_2=f$), $f$ is $\gamma$-integrable with $\int f\,d\gamma=\nu(\mathbb{R}^m)=1$. Since $\nu(A)=\int\mathbf{1}_Af\,d\gamma$ for every Borel $A$, $\nu$ is the measure with density $f$ of claim 3 of Image Measures, Measures with Densities, and Change of Variables.

Let $h$ be bounded Borel and $Z=\int\exp\circ h\,d\gamma>0$ (Claim 1). As $h$ is $\nu$-integrable, claim 3 of Image Measures, Measures with Densities, and Change of Variables shows that $hf$ is $\gamma$-integrable with $\int hf\,d\gamma=\int h\,d\nu$. Put $a(x)=h(x)-\log Z+1$. By The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §young, for every $x$,

the equality by claims 1 and 2 of Basic Properties of the Exponential Function and $\exp(\log Z)=Z$. Both sides are $\gamma$-integrable: $af=hf+(1-\log Z)f$, and $\phi\circ f$, $\exp\circ h$ are integrable, so claim 2 of Linearity and Monotonicity of the Lebesgue Integral applies (linearity and monotonicity) and gives

that is, $\Lambda_h(\nu)\le H(\nu\,|\,\gamma)$. For $h=0$ we have $\int0\,d\nu=0$ and $\exp\circ h=1$, so $\int\exp\circ h\,d\gamma=\gamma(\mathbb{R}^m)=1$ by The Integral of an Indicator Function is the Measure of the Set and $\log1=0$ by (E1); hence $0=\Lambda_0(\nu)\le H(\nu\,|\,\gamma)$.

Claim 3 (Variational criterion). Let $\nu$ and $c$ be as stated.

Step 1 (all bounded Borel test functions). Let $h$ be bounded Borel with bound $b$; we show $\Lambda_h(\nu)\le c$. For Borel $B$ put $\sigma(B)=\nu(B)+\gamma(B)$. Then $\sigma(\varnothing)=0$, and for pairwise disjoint Borel $B_i$ ($i\in\mathbb{N}$) the partial sums of $\sum_i\nu(B_i)$ and of $\sum_i\gamma(B_i)$ are nondecreasing and bounded above by $1$ (claims 1 and 2 of Basic Properties of a Measure), so by countable additivity and the definition of the sum in Measure, Measure Space, and Probability Measure they converge to $\nu(\bigcup_iB_i)$ and $\gamma(\bigcup_iB_i)$; by claim 1 of Arithmetic of Limits of Real Sequences their sums, the partial sums of $\sum_i\sigma(B_i)$, converge to $\sigma(\bigcup_iB_i)$, which is therefore $\sum_i\sigma(B_i)$. So $\sigma$ is a measure on $\mathcal{B}(\mathbb{R}^m)$, the Borel $\sigma$-algebra of $(\mathbb{R}^m,d_E)$ by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §spaces, hence a Borel measure on $(\mathbb{R}^m,d_E)$, with $\sigma(\mathbb{R}^m)=2$; and $\nu(B)\le\sigma(B)$, $\gamma(B)\le\sigma(B)$.

Fix $j\in\mathbb{N}$. The function $h+b$ is Borel (claim 2 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions) with $0\le h+b\le2b$, so Approximation of Measurable Functions by Simple Functions §bounded (in the measure space $(\mathbb{R}^m,\mathcal{B}(\mathbb{R}^m),\sigma)$) gives a nonnegative simple function $s$ with $0\le h(x)+b-s(x)\le2^{-j}$ for every $x$. Let $s=\sum_{i=1}^{r}c_i\mathbf{1}_{A_i}$ be its standard representation: the $A_i$ are Borel, pairwise disjoint and cover $\mathbb{R}^m$. With $a_i=c_i-b$ the function $t=s-b$ equals $\sum_{i=1}^{r}a_i\mathbf{1}_{A_i}$, and $|h(x)-t(x)|\le2^{-j}$ for every $x$. For each $i$, Inner and Outer Regularity of a Finite Borel Measure on a Metric Space, and Lipschitz Approximation of Indicators §lipschitz, applied to $\sigma$, $A_i$ and the positive number $2^{-j}r^{-1}$, gives $g_i:\mathbb{R}^m\to\mathbb{R}$ Lipschitz with some constant $L_i$, with $0\le g_i\le1$, and a Borel set $N_i$ with $\sigma(N_i)\le2^{-j}r^{-1}$ and $g_i=\mathbf{1}_{A_i}$ off $N_i$. Let $N=N_1\cup\dots\cup N_r$; by claim 4 of Basic Properties of a Measure, applied to the sequence $N_1,\dots,N_r,\varnothing,\varnothing,\dots$, $\sigma(N)\le2^{-j}$. Put $u=\sum_{i=1}^{r}a_ig_i$; then $u=t$ off $N$, and by claims 4 and 5 of Properties of the Absolute Value in an Ordered Field and claim 5 of Elementary Arithmetic in an Ordered Field, $|u(x)-u(x')|\le\sum_i|a_i|\,|g_i(x)-g_i(x')|\le\bigl(\sum_i|a_i|L_i\bigr)d_E(x,x')$. Let $\kappa(p)=\max(-b,\min(b,p))$ for real $p$, and write $m_b(p)=\min(b,p)$ and $M_b(p)=\max(-b,p)$, so that $\kappa=M_b\circ m_b$. First, $|m_b(p)-m_b(q)|\le|p-q|$ for all real $p,q$: if $p\le b$ and $q\le b$ the two sides are equal; if $b\le p$ and $b\le q$ the left side is $0$; if $p\le b<q$ the left side is $b-p\le q-p=|p-q|$; and the case $q\le b<p$ is symmetric. Likewise $|M_b(p)-M_b(q)|\le|p-q|$: if $-b\le p$ and $-b\le q$ the two sides are equal; if $p\le-b$ and $q\le-b$ the left side is $0$; if $p<-b\le q$ the left side is $q+b\le q-p=|p-q|$; and the case $q<-b\le p$ is symmetric. Hence $|\kappa(p)-\kappa(q)|\le|m_b(p)-m_b(q)|\le|p-q|$. Next, $-b\le\kappa(p)$ by the definition of the maximum, and $\kappa(p)\le b$ because $-b\le b$ (as $0\le b$) and $m_b(p)\le b$; so $|\kappa(p)|\le b$ by claim 6 of Properties of the Absolute Value in an Ordered Field. Finally, for $p\in[-b,b]$ we have $m_b(p)=p$ as $p\le b$, and then $\kappa(p)=M_b(p)=p$ as $-b\le p$. A map that is Lipschitz with constant $L$, followed by a map $\kappa$ with $|\kappa(p)-\kappa(q)|\le|p-q|$, is again Lipschitz with constant $L$, directly from Lipschitz Map Between Metric Spaces: here $|\kappa(u(x))-\kappa(u(x'))|\le|u(x)-u(x')|\le\bigl(\sum_i|a_i|L_i\bigr)d_E(x,x')$ for all $x,x'$. Hence $h_j=\kappa\circ u$ is Lipschitz with constant $\sum_i|a_i|L_i$ and bounded by $b$, i.e. bounded Lipschitz; it is continuous by A Lipschitz Map is Uniformly Continuous and therefore Borel by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §borel-maps. For $x\notin N$, $|h(x)-h_j(x)|=|\kappa(h(x))-\kappa(u(x))|\le|h(x)-t(x)|\le2^{-j}$; for $x\in N$, $|h(x)-h_j(x)|\le2b$ by claim 5 of Properties of the Absolute Value in an Ordered Field. So $|h-h_j|\le2^{-j}+2b\mathbf{1}_N$, and for $\pi\in\{\nu,\gamma\}$ claim 2 of Linearity and Monotonicity of the Lebesgue Integral and The Integral of an Indicator Function is the Measure of the Set give

Let $Z=\int\exp\circ h\,d\gamma$ and $Z_j=\int\exp\circ h_j\,d\gamma$; both are at least $\exp(-b)$ by (Z), as $|h_j|\le b$. By claim 2 of Linearity and Monotonicity of the Lebesgue Integral and (E2), $|Z-Z_j|\le\int|\exp\circ h-\exp\circ h_j|\,d\gamma\le\exp(b)\int|h-h_j|\,d\gamma\le\exp(b)(1+2b)2^{-j}$, so by (E3) and $\exp(b)\exp(b)=\exp(2b)$, $|\log Z-\log Z_j|\le\exp(2b)(1+2b)2^{-j}$. Also $|\int h\,d\nu-\int h_j\,d\nu|\le\int|h-h_j|\,d\nu\le(1+2b)2^{-j}$ by the same claim. With $C=(1+2b)(1+\exp(2b))$ and the hypothesis applied to the bounded Lipschitz $h_j$,

By Series of Nonnegative Real Numbers, Comparison, and the Geometric Series §geometric (with $r=\tfrac12$) $2^{-j}\to0$, so $c+C2^{-j}\to c$ by claims 1 and 3 of Arithmetic of Limits of Real Sequences, and claim 1 of Order Properties of Limits of Real Sequences, applied with the constant sequence $\Lambda_h(\nu)$, gives $\Lambda_h(\nu)\le c$.

Step 2 (absolute continuity). Let $B$ be Borel with $\gamma(B)=0$, and let $n\in\mathbb{N}$. The function $h=n\mathbf{1}_B$ is bounded Borel, and $\exp\circ h=1+(\exp(n)-1)\mathbf{1}_B$ since $\exp(0)=1$. By claim 2 of Linearity and Monotonicity of the Lebesgue Integral and The Integral of an Indicator Function is the Measure of the Set, $\int\exp\circ h\,d\gamma=1+(\exp(n)-1)\gamma(B)=1$, whose logarithm is $0$ by (E1), and $\int h\,d\nu=n\,\nu(B)$. By Step 1, $n\,\nu(B)=\Lambda_h(\nu)\le c$ for every $n\in\mathbb{N}$. If $\nu(B)>0$, claim 2 of The Archimedean Property of the Real Numbers would give $n$ with $c<n\,\nu(B)$; hence $\nu(B)=0$.

Step 3 (a density). The measure $\gamma$ is finite, and $\sigma$-finite in the sense of Measure, Measure Space, and Probability Measure (take $X_i=\mathbb{R}^m$ for every $i$); $\nu$ is finite; and by Step 2, $\nu(B)=0$ whenever $\gamma(B)=0$. By The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities §existence, $\nu$ has a density $f$ with respect to $\gamma$; by The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities §uniqueness, $f$ is $\gamma$-integrable with $\int f\,d\gamma=1$, and, as in Claim 2, $\nu$ is the measure with density $f$ of claim 3 of Image Measures, Measures with Densities, and Change of Variables.

Step 4 (the entropy bound). For $n\in\mathbb{N}$ let $g_n=\max(f,\exp(-n))$, Borel by claims 1 and 4 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions and positive. By (E1), $\{x:a<\log g_n(x)\}=\{x:\exp(a)<g_n(x)\}$ for real $a$, so $\log\circ g_n$ is Borel by claim 3 of Rational Intervals and Rays Generate the Borel Sigma-Algebra of the Real Line, and $-n\le\log g_n$ since $\exp(-n)\le g_n$. Let $h_n=\min(n,\log\circ g_n)$: Borel by claim 4 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, with $-n\le h_n\le n$, hence bounded Borel. By Step 1, $\Lambda_{h_n}(\nu)\le c$.

Since $h_n\le\log g_n$, (E1) gives $\exp(h_n)\le g_n\le f+\exp(-n)$ pointwise. By claim 2 of Linearity and Monotonicity of the Lebesgue Integral, $Z_n=\int\exp\circ h_n\,d\gamma\le1+\exp(-n)$, so, as $Z_n>0$ by Claim 1, $\log Z_n\le Z_n-1\le\exp(-n)$ by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §log. By claim 3 of Image Measures, Measures with Densities, and Change of Variables, $h_nf$ is $\gamma$-integrable with $\int h_nf\,d\gamma=\int h_n\,d\nu\le c+\log Z_n$, so

Pointwise comparison with $\phi\circ f$, which is Borel by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §continuous: fix $x$ and write $y=f(x)$. If $y\ge1$ then $g_n(x)=y$ (as $\exp(-n)\le\exp(0)=1$), $\log y\ge0$ by (E1), and $h_n(x)y=\min(n,\log y)\,y$, so $0\le h_n(x)y\le\phi(y)$, with equality once $n\ge\log y$. If $\exp(-n)\le y<1$, then $\log y<0$ by (E1), since $y<1=\exp(0)$, and $h_n(x)y=y\log y=\phi(y)$. If $0<y<\exp(-n)$, then $g_n(x)=\exp(-n)$, so $h_n(x)=\min(n,-n)=-n$ because $\log(\exp(-n))=-n$ by The Natural Logarithm, and $\log y<-n$ by (E1), so $\phi(y)=y\log y\le-ny=h_n(x)y\le0$ (claim 5 of Elementary Arithmetic in an Ordered Field). If $y=0$, $h_n(x)y=0=\phi(y)$. Consequently: on $\{f<1\}$, $\phi\circ f\le h_nf\le0$; on $\{f\ge1\}$, $0\le h_nf\le\phi\circ f$ and $h_nf\le h_{n+1}f$; hence $|h_nf|\le|\phi\circ f|$ everywhere. Moreover, by claim 1 of The Archimedean Property of the Real Numbers there is $n_0$ with $|\log y|<n_0$ when $y>0$, and for $n\ge n_0$ the cases above give $h_n(x)y=\phi(y)$ (for $0<y<1$ because then $\exp(-n)\le y$ by (E1)); so $(h_n(x)f(x))_n$ converges to $\phi(f(x))$ for every $x$.

Let $u_n=\max(h_nf,0)$ and $v_n=\max(-h_nf,0)$, Borel by claim 4 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions. On $\{f\ge1\}$, $u_n=h_nf$ and $v_n=0$; on $\{f<1\}$, $u_n=0$ and $0\le v_n\le-\phi\circ f\le\exp(-1)$ by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §lower. Thus $(u_n)$ is nondecreasing with pointwise least upper bound $(\phi\circ f)^{+}$, and $v_n$ is bounded Borel, hence $\gamma$-integrable. By claim 2 of Linearity and Monotonicity of the Lebesgue Integral applied to $u_n=h_nf+v_n$,

using $\exp(-n)\le1$. By Monotone Convergence Theorem, $\int(\phi\circ f)^{+}\,d\gamma\le c+1+\exp(-1)<\infty$; and $(\phi\circ f)^{-}\le\exp(-1)$ by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §lower, so $\int(\phi\circ f)^{-}\,d\gamma\le\exp(-1)$. By the definition of integrability, $\phi\circ f$ is $\gamma$-integrable, so $\nu$ has finite relative entropy with respect to $\gamma$ by Relative Entropy of Probability Measures §relative-entropy, with $H(\nu\,|\,\gamma)=\int\phi\circ f\,d\gamma$.

Since $h_nf\to\phi\circ f$ pointwise and $|h_nf|\le|\phi\circ f|$ with $|\phi\circ f|$ integrable, claim 3 of Dominated Convergence Theorem gives $\int h_nf\,d\gamma\to H(\nu\,|\,\gamma)$. By claim 4 of Basic Properties of the Exponential Function, for every $\varepsilon>0$ there is $M$ with $\exp(u)<\varepsilon$ for $u<-M$, and by claim 1 of The Archimedean Property of the Real Numbers there is $n_1>M$; for $n\ge n_1$ we get $0<\exp(-n)<\varepsilon$, so $\exp(-n)\to0$ and $c+\exp(-n)\to c$ by claim 1 of Arithmetic of Limits of Real Sequences. Claim 1 of Order Properties of Limits of Real Sequences now yields $H(\nu\,|\,\gamma)\le c$.

Claim 4 (Closed sublevel sets). Let $h$ be bounded Lipschitz. It is continuous by A Lipschitz Map is Uniformly Continuous, hence Borel by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §borel-maps, and $Z=\int\exp\circ h\,d\gamma>0$ by Claim 1. For every $n$, Claim 2 gives $\int h\,d\nu_n-\log Z=\Lambda_h(\nu_n)\le H(\nu_n\,|\,\gamma)\le c$. By Probability Measures on Euclidean Space and Random Vectors: Standing Notation §measures the $\nu_n$ and $\nu$ are Borel measures on $(\mathbb{R}^m,d_E)$ of total mass $1$, and $h$ is bounded and continuous, so by the definition of weak convergence $(\int h\,d\nu_n)_n$ converges to $\int h\,d\nu$. Claim 1 of Order Properties of Limits of Real Sequences, applied with the constant sequence $c+\log Z$, gives $\int h\,d\nu\le c+\log Z$, that is, $\Lambda_h(\nu)\le c$. As $h$ was an arbitrary bounded Lipschitz function, Claim 3 shows that $\nu$ has finite relative entropy with respect to $\gamma$ and $H(\nu\,|\,\gamma)\le c$.