Proof of Relative Entropy with Respect to a Diagonal Gaussian Measure on a Hilbert Space: the Moment Bound, the Cutoff Projections, and Bounded, Tight, Weakly Closed, Wasserstein-Closed and Weakly Sequentially Compact Sublevel Sets

Each result cited is universally quantified over the data in its own statement.

Throughout, $\gamma_{c}\in\mathcal{P}(X)$, the set $X$ is nonempty as $0_{X}\in X$, and Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §entropy-inequality and Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §small-sets are used on the measurable space $(X,\mathcal{B}(X))$ with $\gamma=\gamma_{c}$, and Relative Entropy on a Metric Space: the Variational Criterion over Bounded Lipschitz Functions and Sequentially Closed Sublevel Sets under Weak Convergence §closed on the metric space $(X,d)$ with $\gamma=\gamma_{c}$.

Claim 1. Let $\mu\in\mathcal{P}(X)$ have finite relative entropy with respect to $\gamma_{c}$ and put $H=H(\mu\,|\,\gamma_{c})$. For $k\in\mathbb{N}$ the partial sum $s_{k}=\sum_{i=1}^{k}c_{i}$ satisfies $s_{k}\le\bar{c}$ by Series of Nonnegative Real Numbers, Comparison, and the Geometric Series §dominates, the terms being positive, and $c_{k}\le s_{k}$ since the other terms of $s_{k}$ are positive; so $c_{k}\le\bar{c}$. Put $\alpha=1/(4\bar{c})$, a positive real, and $\theta=1/2<1$. Then $2\alpha c_{k}=c_{k}/(2\bar{c})\le1/2=\theta$ for every $k$, so by Moments of a Diagonal Gaussian Measure on a Hilbert Space: Coordinate Covariances, Finite Second Moment, and Exponential Moments of the Squared Norm §exponential the function $x\mapsto\exp(\alpha|x|^{2})$ is Borel and integrable with respect to $\gamma_{c}$, and

$$I=\int_{X}\exp\bigl(\alpha|x|^{2}\bigr)\,\gamma_{c}(dx)\le\exp\Bigl(\frac{\alpha\bar{c}}{1-\theta}\Bigr)=\exp(2\alpha\bar{c})=\exp(1/2).$$

Let $g:X\to\mathbb{R}$, $g(x)=|x|^{2}$; it is nonnegative and Borel, as recorded in the preamble of The Second Moment of a Borel Probability Measure on a Hilbert Space and the Probability Measures with Finite Second Moment, and $\exp\circ(\alpha g)$ is the function $x\mapsto\exp(\alpha|x|^{2})$. By Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §entropy-inequality with $\nu=\mu$ and $t=\alpha$, we get $1\le I$, $g$ is integrable with respect to $\mu$, and $\int_{X}g\,d\mu\le\frac{1}{\alpha}(H+\log I)$. Since $0<I\le\exp(1/2)$ and $\log$ is increasing by claim 2 of Real Powers Through the Exponential, and Elementary Asymptotic Tools: Monotonicity, Null Sequences of Negative Powers, Exponential Domination, Integer Rounding, and Square-Root and Exponential Inequalities, $\log I\le\log\exp(1/2)=1/2$ by The Natural Logarithm; as $1/\alpha=4\bar{c}>0$,

$$\int_{X}g\,d\mu\le4\bar{c}\Bigl(H+\frac12\Bigr)=4\bar{c}\,H+2\bar{c}.$$

Since $g\ge0$, its positive part is $g$ and its negative part is $0$, so by Integrable Function and the Lebesgue Integral the integrability of $g$ means that the integral $M_{2}(\mu)=\int_{X}|x|^{2}\,\mu(dx)\in[0,\infty]$ of The Second Moment of a Borel Probability Measure on a Hilbert Space and the Probability Measures with Finite Second Moment §moment is finite, and the integral of $g$ equals $M_{2}(\mu)$. Hence $\mu\in\mathcal{P}_{2}(X)$ by The Second Moment of a Borel Probability Measure on a Hilbert Space and the Probability Measures with Finite Second Moment §space, and $M_{2}(\mu)\le4\bar{c}\,H(\mu\,|\,\gamma_{c})+2\bar{c}$.

Claim 2. Let $\mu\in\mathcal{K}_{C}$. By claim 1, $\mu\in\mathcal{P}_{2}(X)$ and $M_{2}(\mu)\le4\bar{c}\,H(\mu\,|\,\gamma_{c})+2\bar{c}\le4\bar{c}\,C+2\bar{c}$, since $H(\mu\,|\,\gamma_{c})\le C$ and $4\bar{c}>0$.

Claim 3. For $n\in\mathbb{N}$, $(p_{n})_{\#}\gamma_{c}=\gamma_{c^{(n)}}$ by Diagonal Gaussian Measures on a Hilbert Space §measure. The claims of the projection lemma cited below are used with $\gamma=\gamma_{c}$, so that their $\gamma_{n}=(p_{n})_{\#}\gamma_{c}$ is $\gamma_{c^{(n)}}$. If $\mu$ has finite relative entropy with respect to $\gamma_{c}$, then by Relative Entropy of the Finite-Dimensional Projections of Borel Probability Measures on a Hilbert Space: Monotonicity and Approximation §projections the number $C=H(\mu\,|\,\gamma_{c})$ has the stated property. Conversely, if $C\in\mathbb{R}$ has the stated property, then $\mu$ has finite relative entropy with respect to $\gamma_{c}$ by Relative Entropy of the Finite-Dimensional Projections of Borel Probability Measures on a Hilbert Space: Monotonicity and Approximation §bounded. In that case the sequence $\bigl(H(\mu_{n}\,|\,\gamma_{c^{(n)}})\bigr)_{n}$ is nondecreasing and converges to $H(\mu\,|\,\gamma_{c})$ by Relative Entropy of the Finite-Dimensional Projections of Borel Probability Measures on a Hilbert Space: Monotonicity and Approximation §limit.

Claim 4. Let $C\in\mathbb{R}$ and let $\varepsilon$ be a positive real; the auxiliary numbers are chosen in the order $C'$, $\beta$, $\eta$, $K$. Put $C'=\max\{C,0\}$. Since $\log$ is strictly increasing with $\log1=0$ by claim 2 of Real Powers Through the Exponential, and Elementary Asymptotic Tools: Monotonicity, Null Sequences of Negative Powers, Exponential Domination, Integer Rounding, and Square-Root and Exponential Inequalities, $\log2>0$, so $\beta=(C'+\log2)/\varepsilon$ is positive, and $\exp(\beta)>\exp(0)=1$ by claims 1 and 4 of Basic Properties of the Exponential Function. Put $\eta=1/(\exp(\beta)-1)$, a positive real with $1+1/\eta=\exp(\beta)$, hence $\log(1+1/\eta)=\beta$ by The Natural Logarithm. The metric space $(X,d)$ is complete and separable by Borel Probability Measures on a Real Hilbert Space with an Orthonormal Basis: Standing Notation §space and $\gamma_{c}(X)=1$, so Ulam's Theorem: a Finite Borel Measure on a Complete Separable Metric Space is Tight §tight gives a compact $K\subseteq X$ with $\gamma_{c}(X\setminus K)\le\eta$. Put $A=X\setminus K$, which belongs to $\mathcal{B}(X)$ by Compact Subsets of a Metric Space are Closed and Borel §borel. Let $\mu\in\mathcal{K}_{C}$. If $\gamma_{c}(A)=0$, then $\mu(A)=0\le\varepsilon$ by Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §small-sets. If $\gamma_{c}(A)>0$, then $1+1/\gamma_{c}(A)\ge1+1/\eta=\exp(\beta)$, so $\log(1+1/\gamma_{c}(A))\ge\beta$ by monotonicity of $\log$, and Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §small-sets together with $\mu(A)\ge0$ gives

$$\beta\,\mu(A)\le\mu(A)\log\Bigl(1+\frac{1}{\gamma_{c}(A)}\Bigr)\le H(\mu\,|\,\gamma_{c})+\log2\le C'+\log2=\varepsilon\beta,$$

whence $\mu(A)\le\varepsilon$ as $\beta>0$. Thus $\mu(X\setminus K)\le\varepsilon$ for every $\mu\in\mathcal{K}_{C}$, and $\mathcal{K}_{C}$ is tight by Tight Family of Borel Measures on a Metric Space §tight.

Claim 5. Each $\mu_{j}$ is a Borel measure on $(X,d)$ with $\mu_{j}(X)=1$, of finite relative entropy with respect to $\gamma_{c}$ with $H(\mu_{j}\,|\,\gamma_{c})\le C$, and $\mu_{j}\Rightarrow\mu$ with $\mu\in\mathcal{P}(X)$. By Relative Entropy on a Metric Space: the Variational Criterion over Bounded Lipschitz Functions and Sequentially Closed Sublevel Sets under Weak Convergence §closed with $\nu=\mu$, the measure $\mu$ has finite relative entropy with respect to $\gamma_{c}$ and $H(\mu\,|\,\gamma_{c})\le C$, that is, $\mu\in\mathcal{K}_{C}$.

Claim 6. By claim 2 each $\mu_{j}$ lies in $\mathcal{P}_{2}(X)$, and $\mu\in\mathcal{P}_{2}(X)$ with $W_{2}(\mu_{j},\mu)\to0$, so $\mu_{j}\Rightarrow\mu$ by Wasserstein Convergence on a Hilbert Space: Weak Convergence, Integrals of Continuous Functions of Quadratic Growth, Convergence from Weak Convergence with Uniformly Integrable Second Moments, and Compactness §weak. Since $\mu\in\mathcal{P}(X)$, claim 5 gives $\mu\in\mathcal{K}_{C}$.

Claim 7. Let $(\mu_{j})_{j\in\mathbb{N}}$ be a sequence in $\mathcal{K}_{C}$. Its set of terms is contained in $\mathcal{K}_{C}$, so for every positive real $\varepsilon$ the compact set given for $\mathcal{K}_{C}$ by claim 4 serves for every term, and the sequence is tight by Tight Family of Borel Measures on a Metric Space §sequence. Each $\mu_{j}$ is a Borel measure on $(X,d)$ with $\mu_{j}(X)=1$, so Prokhorov's Theorem on a Metric Space: a Tight Sequence of Borel Probability Measures Has a Weakly Convergent Subsequence §subsequence gives a strictly increasing sequence $(j_{i})_{i\in\mathbb{N}}$ in $\mathbb{N}$ and a Borel measure $\mu$ on $(X,d)$ with $\mu(X)=1$, that is $\mu\in\mathcal{P}(X)$, such that $\mu_{j_{i}}\Rightarrow\mu$. The subsequence $(\mu_{j_{i}})_{i\in\mathbb{N}}$ lies in $\mathcal{K}_{C}$, so claim 5 applied to it gives $\mu\in\mathcal{K}_{C}$.