Proof of The Entropy Inequality for Nonnegative Functions, and Quadratic Moments Bounded by the Relative Entropy with Respect to a Diagonal Gaussian Measure

Each result cited below is universally quantified over the data in its own statement.

Step 0 (Two conventions on integrals). Let $(X,\mathcal{F},\mu)$ be a measure space and $f:X\to\mathbb{R}$ measurable with $0\le f(x)$ for every $x$. The integral of the zero function is $0$: it equals $\int_{X}0\cdot f\,d\mu=0\cdot\int_{X}f\,d\mu=0$ by Linearity and Monotonicity of the Lebesgue Integral §nonnegative with the constant $0$ and the convention $0\cdot\infty=0$. Since $f^{+}=f$ and $f^{-}=0$ in the notation of Integrable Function and the Lebesgue Integral, that definition shows that $f$ is integrable exactly when its integral as a $[0,\infty]$-valued map is finite, and that its integral as an integrable function then equals that $[0,\infty]$-integral. We use this silently for the nonnegative functions below, which is consistent with the reading of integrals of nonnegative Borel functions fixed in Probability Measures on Euclidean Space and Random Vectors: Standing Notation §measures.

Step 1 (Claim 1: truncation). Let $\gamma,\nu$ and $h$ be as in claim 1. For $n\in\mathbb{N}$ let $h_{n}=\min(h,n)$, the pointwise minimum of $h$ and the constant function $n$. The constant function is Borel by claim 1 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions, so $h_{n}$ is Borel by claim 4 of that lemma, applied on the measurable space $(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}))$. Since $h\ge0$ we have $0\le h_{n}(x)\le n$ for every $x$, so $h_{n}$ is bounded with bound $n$. Moreover $h_{n}(x)\le h_{n+1}(x)\le h(x)$ for every $x$ and $n$, since $n\le n+1$; and for each $x$ there is, by claim 1 of The Archimedean Property of the Real Numbers, an $n\in\mathbb{N}$ with $h(x)<n$, for which $h_{n}(x)=h(x)$; hence $\sup_{n}h_{n}(x)=h(x)$ for every $x\in\mathbb{R}^{d}$.

Step 2 (Claim 1: the exponential moment is a positive real number). By hypothesis $\exp\circ h$ is integrable with respect to $\gamma$; it is nonnegative by claim 2 of Basic Properties of the Exponential Function, so by Step 0 the number $E=\int_{\mathbb{R}^{d}}\exp\circ h\,d\gamma$ is finite. For each $n$, Relative Entropy on Euclidean Space: the Gibbs Inequality, a Variational Criterion for Finite Relative Entropy, and Closed Sublevel Sets under Weak Convergence §functional with $m=d$, the measure $\gamma$ and the bounded Borel function $h_{n}$ shows that $\exp\circ h_{n}$ is Borel and that $E_{n}=\int_{\mathbb{R}^{d}}\exp\circ h_{n}\,d\gamma$ is a positive real number. As $\exp$ is increasing by claim 4 of Basic Properties of the Exponential Function and $h_{n}\le h$, we have $\exp\circ h_{n}\le\exp\circ h$ pointwise, so $E_{n}\le E$ by the monotonicity in Linearity and Monotonicity of the Lebesgue Integral §nonnegative. Hence $0<E_{n}\le E<\infty$, and $E$ is a positive real number. Moreover $\log E_{n}\le\log E$: otherwise $\log E<\log E_{n}$, and since $\exp$ is strictly increasing (claim 4 of Basic Properties of the Exponential Function) and inverse to $\log$ (The Natural Logarithm), $E=\exp(\log E)<\exp(\log E_{n})=E_{n}$, a contradiction.

Step 3 (Claim 1: the inequality). Since $\nu$ has finite relative entropy with respect to $\gamma$, $H(\nu\,|\,\gamma)$ is a real number by Relative Entropy of Probability Measures §relative-entropy. For each $n$, the Gibbs inequality Relative Entropy on Euclidean Space: the Gibbs Inequality, a Variational Criterion for Finite Relative Entropy, and Closed Sublevel Sets under Weak Convergence §gibbs, applied with $m=d$, the measures $\gamma,\nu$ and the bounded Borel function $h_{n}$, gives

$$\int_{\mathbb{R}^{d}}h_{n}\,d\nu-\log E_{n}\le H(\nu\,|\,\gamma),\qquad\text{hence}\qquad\int_{\mathbb{R}^{d}}h_{n}\,d\nu\le H(\nu\,|\,\gamma)+\log E_{n}\le H(\nu\,|\,\gamma)+\log E$$

by Step 2. By Step 1 the Borel functions $h_{n}:\mathbb{R}^{d}\to[0,\infty)$ are nondecreasing in $n$ with pointwise supremum $h$, so Monotone Convergence Theorem applied to $(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}),\nu)$ gives $\int_{\mathbb{R}^{d}}h\,d\nu=\sup_{n}\int_{\mathbb{R}^{d}}h_{n}\,d\nu$ in $[0,\infty]$. Each term of the supremum is at most the real number $H(\nu\,|\,\gamma)+\log E$, hence so is the supremum. Thus $\int_{\mathbb{R}^{d}}h\,d\nu\le H(\nu\,|\,\gamma)+\log\int_{\mathbb{R}^{d}}\exp\circ h\,d\gamma<\infty$, and by Step 0 the nonnegative Borel function $h$ is integrable with respect to $\nu$. Together with Step 2 this proves claim 1.

Step 4 (Claim 2: the quadratic form and its exponential moment). Let $c,\gamma_{c},w,t$ and $\mu$ be as in claim 2, and put $q(x)=\sum_{i=1}^{d}w_{i}x_{i}^{2}$ and $h=tq$, so that $h(x)=\sum_{i=1}^{d}(tw_{i})x_{i}^{2}$. Points of $\mathbb{R}^{d}$ are tuples by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §spaces, and each coordinate map $x\mapsto x_{i}$ is Borel by claims 1 and 5 of The Borel Sigma-Algebra of a Euclidean Space as a Product, and Measurability of Projections, Sequentially Continuous Maps, and Open and Closed Sets; hence $q$ and $h$ are Borel by claims 2 and 3 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions. As $t>0$, $w_{i}\ge0$ and $x_{i}^{2}\ge0$, both take values in $[0,\infty)$. The map $\exp$ is differentiable at every point of $\mathbb{R}$ by claim 3 of Basic Properties of the Exponential Function; $\mathbb{R}$ is an interval and each $s\in\mathbb{R}$ is an interior point of it, as $s-1<s<s+1$, so Differentiability at an Interior Point Implies Continuity There with $I=\mathbb{R}$ shows that $\exp$ is continuous at every point of $(\mathbb{R},d_{\mathbb{R}})$, hence Borel, and $\exp\circ h$ is Borel as a composition of Borel maps, both by Probability Measures on Euclidean Space and Random Vectors: Standing Notation §borel-maps. Put $t_{i}=tw_{i}$ and $u_{i}=2t_{i}c_{i}=2tw_{i}c_{i}$ for $i\in[d]$; since $t>0$, $w_{i}\ge0$, $c_{i}>0$ (The Diagonal Gaussian Density on Euclidean Space and Its Notation §variances) and $4tw_{i}c_{i}\le1$, we have $0\le u_{i}\le\tfrac12<1$. The density $\rho_{c}$ is positive and Borel by The Diagonal Gaussian Density on Euclidean Space: Regularity, Gradient, Normalization, Second Moments and Exponential Moments of Diagonal Quadratic Forms §regularity, and $\gamma_{c}$ is the measure with density $\rho_{c}$ with respect to $\lambda_{d}$ and a member of $\mathcal{P}(\mathbb{R}^{d})$ by Diagonal Gaussian Measures on Euclidean Space §measure. Claim 3 of Image Measures, Measures with Densities, and Change of Variables, applied to the measure space $(\mathbb{R}^{d},\mathcal{B}(\mathbb{R}^{d}),\lambda_{d})$, the density $\rho_{c}$ and the Borel function $\exp\circ h$, gives $\int_{\mathbb{R}^{d}}\exp\circ h\,d\gamma_{c}=\int_{\mathbb{R}^{d}}(\exp\circ h)\rho_{c}\,d\lambda_{d}$ in $[0,\infty]$. By The Diagonal Gaussian Density on Euclidean Space: Regularity, Gradient, Normalization, Second Moments and Exponential Moments of Diagonal Quadratic Forms §exponential with the variance vector $c$ and the vector $(t_{1},\dots,t_{d})$, which satisfies $2t_{i}c_{i}=u_{i}<1$, the nonnegative function $(\exp\circ h)\rho_{c}$ is integrable with respect to $\lambda_{d}$ with integral $\exp\bigl(-\tfrac12\sum_{i=1}^{d}\log(1-u_{i})\bigr)$. By Step 0, $\exp\circ h$ is integrable with respect to $\gamma_{c}$ and

$$\log\int_{\mathbb{R}^{d}}\exp\circ h\,d\gamma_{c}=-\tfrac12\sum_{i=1}^{d}\log(1-u_{i}),$$

using $\log(\exp(a))=a$ from The Natural Logarithm.

Step 5 (Claim 2: an elementary bound on the logarithm). Let $u\in\mathbb{R}$ with $0\le u\le\tfrac12$. Then $s=1-u$ satisfies $\tfrac12\le s\le1$, so $s$ is positive, and the lower bound in The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §log applied to $s$ gives $\log(1-u)\ge1-\tfrac{1}{1-u}=-\tfrac{u}{1-u}$. Since $\tfrac{1}{1-u}\le2$ and $u\ge0$, we get $\tfrac{u}{1-u}\le2u$, hence $-\tfrac12\log(1-u)\le\tfrac12\cdot\tfrac{u}{1-u}\le u$. Applying this to each $u_{i}$ of Step 4 and summing over $i\in[d]$,

$$-\tfrac12\sum_{i=1}^{d}\log(1-u_{i})\le\sum_{i=1}^{d}u_{i}=2t\sum_{i=1}^{d}w_{i}c_{i}.$$

Step 6 (Claim 2: conclusion). The measure $\mu\in\mathcal{P}(\mathbb{R}^{d})$ has finite relative entropy with respect to $\gamma_{c}\in\mathcal{P}(\mathbb{R}^{d})$, and $h$ is a Borel map into $[0,\infty)$ with $\exp\circ h$ integrable with respect to $\gamma_{c}$ (Step 4). Claim 1 (Steps 1 to 3), applied with $\gamma=\gamma_{c}$, $\nu=\mu$ and this $h$, shows that $h$ is integrable with respect to $\mu$ and, with Steps 4 and 5,

$$\int_{\mathbb{R}^{d}}h\,d\mu\le H(\mu\,|\,\gamma_{c})-\tfrac12\sum_{i=1}^{d}\log(1-u_{i})\le H(\mu\,|\,\gamma_{c})+2t\sum_{i=1}^{d}w_{i}c_{i}.$$

Since $q=\tfrac1t h$ is the scalar multiple of the $\mu$-integrable function $h$ by the real number $\tfrac1t$, it is integrable with respect to $\mu$ and $\int_{\mathbb{R}^{d}}q\,d\mu=\tfrac1t\int_{\mathbb{R}^{d}}h\,d\mu$ by Linearity and Monotonicity of the Lebesgue Integral §integrable (with $f=g=h$, $a=\tfrac1t$ and $b=0$). Multiplying the last display by the positive number $\tfrac1t$ gives $\int_{\mathbb{R}^{d}}\sum_{i=1}^{d}w_{i}x_{i}^{2}\,\mu(dx)\le\tfrac1t H(\mu\,|\,\gamma_{c})+2\sum_{i=1}^{d}w_{i}c_{i}$, which is claim 2.

Step 7 (Claim 3). Let $c$, $\gamma_{c}$ and $\mu$ be as in claim 3. The number $c_{\max}$ is positive by The Diagonal Gaussian Density on Euclidean Space and Its Notation §variances, so $t=\tfrac{1}{4c_{\max}}$ is positive; take $w_{i}=1$ for every $i\in[d]$. Then $4tw_{i}c_{i}=c_{i}/c_{\max}\le1$, since $c_{i}\le c_{\max}$. By claim 2 (Steps 4 to 6) with these $w$ and $t$, the function $x\mapsto\sum_{i=1}^{d}x_{i}^{2}$ is integrable with respect to $\mu$ and its integral is at most $4c_{\max}H(\mu\,|\,\gamma_{c})+2\sum_{i=1}^{d}c_{i}$. By Elementary Properties of the Euclidean Norm on $\mathbb{R}^n$ §square, $\sum_{i=1}^{d}x_{i}^{2}=\lVert x\rVert^{2}$ for every $x\in\mathbb{R}^{d}$. Hence, by The Second Moment of a Probability Measure on Euclidean Space and the Probability Measures with Finite Second Moment §moment and Step 0, $M_{2}(\mu)=\int_{\mathbb{R}^{d}}\lVert x\rVert^{2}\,\mu(dx)\le4c_{\max}H(\mu\,|\,\gamma_{c})+2\sum_{i=1}^{d}c_{i}<\infty$, and $\mu\in\mathcal{P}_{2}(\mathbb{R}^{d})$ by The Second Moment of a Probability Measure on Euclidean Space and the Probability Measures with Finite Second Moment §space. This proves claim 3.