Proof of Relative Entropy of the Finite-Dimensional Projections of Borel Probability Measures on a Hilbert Space: Monotonicity and Approximation

Each result cited is universally quantified over the data in its own statement.

Throughout, push-forwards are the image measures of claim 1 of Image Measures, Measures with Densities, and Change of Variables, as fixed in Borel Probability Measures on a Real Hilbert Space with an Orthonormal Basis: Standing Notation §pushforward; by that claim $\mu_{n}$ and $\gamma_{n}$ are probability measures on $(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n}))$, and the image measures $T_{\#}\nu$ of Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §data-processing are the same objects.

Claim 1. Let $\pi:\mathbb{R}^{n}\to\mathbb{R}^{m}$ be the map $\pi(y)=(y_{1},\dots,y_{m})$. For $x\in X$ the definition of the coordinate maps in Borel Probability Measures on a Real Hilbert Space with an Orthonormal Basis: Standing Notation §coordinates gives $p_{m}(x)=(x_{1},\dots,x_{m})=\pi(p_{n}(x))$, so $p_{m}=\pi\circ p_{n}$. For $y\in\mathbb{R}^{n}$, $p_{n}(p_{n}^{*}(y))=y$ by Borel Sets of a Hilbert Space with an Orthonormal Basis: Coordinates, Determination by Finite-Dimensional Projections, and Pairs §continuity, hence $p_{m}(p_{n}^{*}(y))=\pi(p_{n}(p_{n}^{*}(y)))=\pi(y)$, so $\pi=p_{m}\circ p_{n}^{*}$. The maps $p_{n}^{*}:(\mathbb{R}^{n},d_{E})\to(X,d)$ and $p_{m}:(X,d)\to(\mathbb{R}^{m},d_{E})$ are Borel by Borel Sets of a Hilbert Space with an Orthonormal Basis: Coordinates, Determination by Finite-Dimensional Projections, and Pairs §continuity, so $\pi$ is measurable with respect to $\mathcal{B}(\mathbb{R}^{n})$ and $\mathcal{B}(\mathbb{R}^{m})$ by claim 4 of Borel Measurability and Bounded Integration on a Metric Space. For $B\in\mathcal{B}(\mathbb{R}^{m})$ we have $p_{n}^{-1}(\pi^{-1}(B))=(\pi\circ p_{n})^{-1}(B)=p_{m}^{-1}(B)$, hence

$$(\pi_{\#}\mu_{n})(B)=\mu_{n}(\pi^{-1}(B))=\mu\bigl(p_{n}^{-1}(\pi^{-1}(B))\bigr)=\mu\bigl(p_{m}^{-1}(B)\bigr)=\mu_{m}(B),$$

so $\pi_{\#}\mu_{n}=\mu_{m}$, and in the same way $\pi_{\#}\gamma_{n}=\gamma_{m}$. Now apply Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §data-processing with $(S,\mathcal{S})=(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n}))$, $\nu=\mu_{n}$, $\gamma=\gamma_{n}$, $(S',\mathcal{S}')=(\mathbb{R}^{m},\mathcal{B}(\mathbb{R}^{m}))$ and $T=\pi$: since $\mu_{n}$ has finite relative entropy with respect to $\gamma_{n}$, the measure $\mu_{m}=\pi_{\#}\mu_{n}$ has finite relative entropy with respect to $\gamma_{m}=\pi_{\#}\gamma_{n}$, and $H(\mu_{m}\,|\,\gamma_{m})\le H(\mu_{n}\,|\,\gamma_{n})$.

Claim 2. Let $n\in\mathbb{N}$. The map $p_{n}$ is measurable with respect to $\mathcal{B}(X)$ and $\mathcal{B}(\mathbb{R}^{n})$ by Borel Sets of a Hilbert Space with an Orthonormal Basis: Coordinates, Determination by Finite-Dimensional Projections, and Pairs §continuity, and $(p_{n})_{\#}\mu=\mu_{n}$, $(p_{n})_{\#}\gamma=\gamma_{n}$. Applying Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §data-processing with $(S,\mathcal{S})=(X,\mathcal{B}(X))$, $\nu=\mu$, $\gamma=\gamma$, $(S',\mathcal{S}')=(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n}))$ and $T=p_{n}$ gives that $\mu_{n}$ has finite relative entropy with respect to $\gamma_{n}$ and $H(\mu_{n}\,|\,\gamma_{n})\le H(\mu\,|\,\gamma)$.

Claim 3. The set $X$ is nonempty, as $0_{X}\in X$, and $\mu,\gamma$ are Borel measures on $(X,d)$ of total mass $1$; likewise, for each $n$, $\mathbb{R}^{n}$ is nonempty, its Borel $\sigma$-algebra for $d_{E}$ is $\mathcal{B}(\mathbb{R}^{n})$ by Borel Probability Measures on a Real Hilbert Space with an Orthonormal Basis: Standing Notation §measures, and $\mu_{n},\gamma_{n}$ are Borel measures on $(\mathbb{R}^{n},d_{E})$ of total mass $1$. So the standing data of Relative Entropy on a Metric Space: the Variational Criterion over Bounded Lipschitz Functions and Sequentially Closed Sublevel Sets under Weak Convergence §lipschitz and Relative Entropy on a Metric Space: the Variational Criterion over Bounded Lipschitz Functions and Sequentially Closed Sublevel Sets under Weak Convergence §criterion may be taken to be $(X,d)$ with $\nu=\mu$ and $\gamma=\gamma$, or $(\mathbb{R}^{n},d_{E})$ with $\nu=\mu_{n}$ and $\gamma=\gamma_{n}$. Let $h:X\to\mathbb{R}$ be bounded Lipschitz in the sense of that lemma for $(X,d)$, and fix nonnegative reals $M$ and $L$ with $|h(x)|\le M$ and $|h(x)-h(x')|\le L\,|x-x'|$ for all $x,x'\in X$. Fix $n\in\mathbb{N}$ and put $g_{n}=h\circ p_{n}^{*}:\mathbb{R}^{n}\to\mathbb{R}$ and $f_{n}=h\circ P_{n}:X\to\mathbb{R}$. The maps $p_{n}^{*}$ and $P_{n}$ are Lipschitz with constant $1$ by Borel Sets of a Hilbert Space with an Orthonormal Basis: Coordinates, Determination by Finite-Dimensional Projections, and Pairs §continuity, so for $y,y'\in\mathbb{R}^{n}$ and $x,x'\in X$

$$|g_{n}(y)|\le M,\quad |g_{n}(y)-g_{n}(y')|\le L\,|p_{n}^{*}(y)-p_{n}^{*}(y')|\le L\,\lVert y-y'\rVert,\quad |f_{n}(x)|\le M,\quad |f_{n}(x)-f_{n}(x')|\le L\,|P_{n}x-P_{n}x'|\le L\,|x-x'| .$$

Hence $g_{n}$ is bounded Lipschitz on $(\mathbb{R}^{n},d_{E})$ and $f_{n}$ is bounded Lipschitz on $(X,d)$, and both are bounded measurable by Relative Entropy on a Metric Space: the Variational Criterion over Bounded Lipschitz Functions and Sequentially Closed Sublevel Sets under Weak Convergence §lipschitz; so is $h$. By Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §functional, $g_{n}$, $f_{n}$, $h$, $\exp\circ g_{n}$, $\exp\circ f_{n}$ and $\exp\circ h$ are integrable with respect to every probability measure on the respective space, $\exp\circ g_{n}$ and $\exp\circ f_{n}$ are bounded measurable, and $\int_{\mathbb{R}^{n}}\exp\circ g_{n}\,d\gamma_{n}$, $\int_{X}\exp\circ f_{n}\,d\gamma$ and $b=\int_{X}\exp\circ h\,d\gamma$ are positive real numbers. Since $P_{n}=p_{n}^{*}\circ p_{n}$ by Borel Probability Measures on a Real Hilbert Space with an Orthonormal Basis: Standing Notation §coordinates, we have $f_{n}=g_{n}\circ p_{n}$ and $\exp\circ f_{n}=(\exp\circ g_{n})\circ p_{n}$, and claim 2 of Image Measures, Measures with Densities, and Change of Variables, applied to the measure spaces $(X,\mathcal{B}(X),\mu)$ and $(X,\mathcal{B}(X),\gamma)$ with $T=p_{n}$, gives

$$\int_{\mathbb{R}^{n}}g_{n}\,d\mu_{n}=\int_{X}f_{n}\,d\mu,\qquad \int_{\mathbb{R}^{n}}\exp\circ g_{n}\,d\gamma_{n}=\int_{X}\exp\circ f_{n}\,d\gamma .$$

Therefore, with the numbers $\Lambda$ of Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §lambda, Relative Entropy on a Measurable Space: the Gibbs Inequality, the Variational Criterion and Formula, the Entropy Inequality, Small Sets and Data Processing §gibbs on $(\mathbb{R}^{n},\mathcal{B}(\mathbb{R}^{n}))$ with $\nu=\mu_{n}$, $\gamma=\gamma_{n}$ and the bounded measurable $g_{n}$, together with the hypothesis, gives

$$\Lambda^{\gamma}_{f_{n}}(\mu)=\Lambda^{\gamma_{n}}_{g_{n}}(\mu_{n})\le H(\mu_{n}\,|\,\gamma_{n})\le C\qquad(n\in\mathbb{N}).$$

Now let $x\in X$. The sequence $(X_{n})_{n\in\mathbb{N}}$ is exhausting and $P_{n}$ is the orthogonal projection onto $X_{n}$ by Borel Probability Measures on a Real Hilbert Space with an Orthonormal Basis: Standing Notation §coordinates, so $(P_{n}x)_{n\in\mathbb{N}}$ converges to $x$ in $(X,d)$ by Exhausting Sequences of Finite-Dimensional Subspaces in a Separable Real Hilbert Space, and Their Projections §tail (with $H=X$ and $H_{n}=X_{n}$). As $|f_{n}(x)-h(x)|\le L\,|P_{n}x-x|$, the sequence $(f_{n}(x))_{n}$ converges to $h(x)$, and then $(\exp(f_{n}(x)))_{n}$ converges to $\exp(h(x))$ by claim 3(f) of Real Powers Through the Exponential, and Elementary Asymptotic Tools: Monotonicity, Null Sequences of Negative Powers, Exponential Domination, Integer Rounding, and Square-Root and Exponential Inequalities. Moreover $|f_{n}(x)|\le M$ and $0<\exp(f_{n}(x))\le\exp(M)$ by claims 2 and 4 of Basic Properties of the Exponential Function. The constant functions $M$ and $\exp(M)$ are integrable with respect to $\mu$ and $\gamma$ by claim 6(a) of Borel Measurability and Bounded Integration on a Metric Space. Hence Dominated Convergence Theorem, applied on $(X,\mathcal{B}(X),\mu)$ to $(f_{n})_{n}$ and on $(X,\mathcal{B}(X),\gamma)$ to $(\exp\circ f_{n})_{n}$, gives

$$a_{n}=\int_{X}f_{n}\,d\mu\to\int_{X}h\,d\mu,\qquad b_{n}=\int_{X}\exp\circ f_{n}\,d\gamma\to b .$$

Next, $\log b_{n}\to\log b$. Indeed, for positive reals $s,t$ we have $\log s-\log t=\log(s/t)$ because $\log(st)=\log s+\log t$ by The Natural Logarithm, and The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §log gives $(s-t)/s=1-t/s\le\log(s/t)\le s/t-1=(s-t)/t$, so $|\log s-\log t|\le|s-t|/\min\{s,t\}$. Since $b>0$ and $b_{n}\to b$, there is $N\in\mathbb{N}$ with $b_{n}\ge b/2$ for $n\ge N$, and then $|\log b_{n}-\log b|\le 2|b_{n}-b|/b\to0$. Consequently $\Lambda^{\gamma}_{f_{n}}(\mu)=a_{n}-\log b_{n}$ converges to $\int_{X}h\,d\mu-\log b=\Lambda^{\gamma}_{h}(\mu)$, and since every term is at most $C$, $\Lambda^{\gamma}_{h}(\mu)\le C$. As $h$ was an arbitrary bounded Lipschitz function on $(X,d)$, Relative Entropy on a Metric Space: the Variational Criterion over Bounded Lipschitz Functions and Sequentially Closed Sublevel Sets under Weak Convergence §criterion on $(X,d)$ with $\nu=\mu$ shows that $\mu$ has finite relative entropy with respect to $\gamma$ and $H(\mu\,|\,\gamma)\le C$.

Claim 4. Let $\mu$ have finite relative entropy with respect to $\gamma$ and write $H_{n}=H(\mu_{n}\,|\,\gamma_{n})$, defined for every $n$ by claim 2, which also gives $H_{n}\le H(\mu\,|\,\gamma)$. Claim 1 with $m=n$ and $n+1$ in place of $n$ gives $H_{n}\le H_{n+1}$, so $(H_{n})_{n}$ is nondecreasing. The set $\{H_{n}:n\in\mathbb{N}\}$ is nonempty and bounded above by $H(\mu\,|\,\gamma)$, so it has a least upper bound $\Theta\le H(\mu\,|\,\gamma)$. Given a real $\varepsilon>0$ there is $N$ with $\Theta-\varepsilon<H_{N}$, and then $\Theta-\varepsilon<H_{N}\le H_{n}\le\Theta$ for all $n\ge N$; so $H_{n}\to\Theta$. Every $\mu_{n}$ has finite relative entropy with respect to $\gamma_{n}$ and $H_{n}\le\Theta$, so claim 3 with $C=\Theta$ gives $H(\mu\,|\,\gamma)\le\Theta$. Hence $\Theta=H(\mu\,|\,\gamma)$, and $(H_{n})_{n}$ is nondecreasing and converges to $H(\mu\,|\,\gamma)$.