Proof of Relative Entropy of the Image under a Bimeasurable Bijection that Moves the Reference Measure by a Positive Density

Each result cited is universally quantified over the data in its own statement.

Throughout, write $\lambda=T_{\#}\gamma$ and let $\phi$ be the function $s\mapsto s\log s$ of The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm, so that $\phi(0)=0$ and $\phi(t)=t\log t$ for positive $t$. By claim 1 of Image Measures, Measures with Densities, and Change of Variables, applied to the measure space $(S,\mathcal{S},\gamma)$, respectively $(S,\mathcal{S},\nu)$, and the measurable map $T$, both $\lambda$ and $T_{\#}\nu$ are probability measures on $(S,\mathcal{S})$, in particular finite. By the convention of The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities, the hypothesis that $g$ is a density of $\lambda$ with respect to $\gamma$ says that $\lambda$ is the measure with density $g$ with respect to $\gamma$ of claim 3 of Image Measures, Measures with Densities, and Change of Variables; likewise, if $f$ is a density of $\nu$ with respect to $\gamma$, then $\nu$ is the measure with density $f$ with respect to $\gamma$. We record the two transfer rules used below.

(E1) By claim 3 of Image Measures, Measures with Densities, and Change of Variables (measure space $(S,\mathcal{S},\gamma)$, density $g$): for every measurable $u:S\to[0,\infty]$ one has $\int_{S}u\,d\lambda=\int_{S}u\,g\,d\gamma$; and a measurable $u:S\to\mathbb{R}$ is integrable with respect to $\lambda$ if and only if $ug$ is integrable with respect to $\gamma$, in which case $\int_{S}u\,d\lambda=\int_{S}u\,g\,d\gamma$ in $\mathbb{R}$.

(E2) By claim 2 of Image Measures, Measures with Densities, and Change of Variables (change of variables, with the measure space $(S,\mathcal{S},\gamma)$ and the map $T$, whose image measure is $\lambda$): for every measurable $u:S\to[0,\infty]$ one has $\int_{S}u\,d\lambda=\int_{S}u\circ T\,d\gamma$; and a measurable $u:S\to\mathbb{R}$ is integrable with respect to $\lambda$ if and only if $u\circ T$ is integrable with respect to $\gamma$, in which case $\int_{S}u\,d\lambda=\int_{S}u\circ T\,d\gamma$ in $\mathbb{R}$.

Step 0 (Composition with $T^{-1}$). Let $u:S\to\mathbb{R}$ be measurable. For every Borel set $E\subseteq\mathbb{R}$, the preimage of $E$ under $u\circ T^{-1}$ is the preimage under the map $T^{-1}$ of the set $u^{-1}(E)\in\mathcal{S}$, and it belongs to $\mathcal{S}$ because $T^{-1}$ is measurable with respect to $\mathcal{S}$; so $u\circ T^{-1}$ is measurable. Moreover $(u\circ T^{-1})\circ T=u$, since $T^{-1}(T(s))=s$ for every $s\in S$.

Step 1 (Claim 1). Let $f$ be a density of $\nu$ with respect to $\gamma$; thus $f$ is measurable and $0\le f(s)$ for every $s\in S$. By Step 0 the function $f\circ T^{-1}$ is measurable and nonnegative, and $F=(f\circ T^{-1})\,g$ is measurable by claim 3 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions and nonnegative because $0<g$. Fix $B\in\mathcal{S}$ and put $u_{B}=\mathbf{1}_{B}\,(f\circ T^{-1})$, a measurable function with values in $[0,\infty)$ by claims 1 and 3 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions. Since $u_{B}\,g=\mathbf{1}_{B}F$, rule (E1) gives $\int_{S}\mathbf{1}_{B}F\,d\gamma=\int_{S}u_{B}\,d\lambda$, and rule (E2) gives $\int_{S}u_{B}\,d\lambda=\int_{S}u_{B}\circ T\,d\gamma$. For $s\in S$ one has $(u_{B}\circ T)(s)=\mathbf{1}_{B}(T(s))\,f(s)=\mathbf{1}_{T^{-1}(B)}(s)\,f(s)$ by Step 0, and $T^{-1}(B)\in\mathcal{S}$ because $T$ is measurable. As $f$ is a density of $\nu$ with respect to $\gamma$,

$$\int_{S}\mathbf{1}_{B}F\,d\gamma=\int_{S}\mathbf{1}_{T^{-1}(B)}\,f\,d\gamma=\nu\bigl(T^{-1}(B)\bigr)=(T_{\#}\nu)(B).$$

This holds for every $B\in\mathcal{S}$, so $F$ is a density of the finite measure $T_{\#}\nu$ with respect to $\gamma$ in the sense of The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities. This proves claim 1.

Step 2 (Set-up for claim 2). By Relative Entropy of Probability Measures §relative-entropy there is a density $f$ of $\nu$ with respect to $\gamma$ such that $\phi\circ f$ is integrable with respect to $\gamma$, and $H(\nu\,|\,\gamma)=\int_{S}\phi\circ f\,d\gamma$. Fix such an $f$ and let $F=(f\circ T^{-1})\,g$, a density of $T_{\#}\nu$ with respect to $\gamma$ by Step 1. Let $\ell=\log\circ g:S\to\mathbb{R}$, defined because $0<g$. By hypothesis $\ell\circ T$ is integrable with respect to $\nu$, hence in particular measurable, and then $\ell=(\ell\circ T)\circ T^{-1}$, because $T(T^{-1}(s))=s$ for every $s\in S$ ($T^{-1}$ being the inverse of the bijection $T$), so $\ell$ is measurable by Step 0 applied to $u=\ell\circ T$.

Step 3 (Pointwise splitting). Let $\psi_{1}=\phi\circ f\circ T^{-1}$ and $\psi_{2}=(f\circ T^{-1})\,\ell$. The function $\phi\circ f$ is measurable by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §continuous, so $\psi_{1}$ is measurable by Step 0, and $\psi_{2}$ is measurable by Step 0, Step 2 and claim 3 of Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions. We claim $\phi\circ F=\psi_{1}g+\psi_{2}g$. Fix $s\in S$ and put $\alpha=f(T^{-1}(s))$ and $\beta=g(s)$, so $0\le\alpha$ and $0<\beta$. If $\alpha=0$ both $\phi(\alpha\beta)$ and $\beta\,\phi(\alpha)+\alpha\beta\log\beta$ equal $0$, since $\phi(0)=0$. If $0<\alpha$, then $0<\alpha\beta$ and, using $\log(\alpha\beta)=\log\alpha+\log\beta$ from The Natural Logarithm,

$$\phi(\alpha\beta)=\alpha\beta\log(\alpha\beta)=\beta\,(\alpha\log\alpha)+\alpha\beta\log\beta=\beta\,\phi(\alpha)+\alpha\beta\log\beta .$$

In both cases $\phi(F(s))=\psi_{1}(s)g(s)+\psi_{2}(s)g(s)$.

Step 4 (First term). By Step 0, $\psi_{1}\circ T=\phi\circ f$, which is integrable with respect to $\gamma$. By rule (E2), $\psi_{1}$ is integrable with respect to $\lambda$ and $\int_{S}\psi_{1}\,d\lambda=\int_{S}\phi\circ f\,d\gamma=H(\nu\,|\,\gamma)$. By rule (E1), $\psi_{1}g$ is integrable with respect to $\gamma$ and $\int_{S}\psi_{1}g\,d\gamma=H(\nu\,|\,\gamma)$.

Step 5 (Second term). Since $\nu$ is the measure with density $f$ with respect to $\gamma$, claim 3 of Image Measures, Measures with Densities, and Change of Variables (measure space $(S,\mathcal{S},\gamma)$, density $f$, real-valued measurable function $\ell\circ T$, which is integrable with respect to $\nu$) shows that $(\ell\circ T)\,f$ is integrable with respect to $\gamma$ and $\int_{S}(\ell\circ T)\,f\,d\gamma=\int_{S}\ell\circ T\,d\nu$. By Step 0, $\psi_{2}\circ T=\bigl((f\circ T^{-1})\circ T\bigr)(\ell\circ T)=f\,(\ell\circ T)$. Hence rule (E2) shows that $\psi_{2}$ is integrable with respect to $\lambda$ with $\int_{S}\psi_{2}\,d\lambda=\int_{S}\ell\circ T\,d\nu$, and rule (E1) shows that $\psi_{2}g$ is integrable with respect to $\gamma$ with

$$\int_{S}\psi_{2}g\,d\gamma=\int_{S}\ell\circ T\,d\nu=\int_{S}\log g(T(s))\,\nu(ds).$$

Step 6 (Conclusion). By Step 3 and Linearity and Monotonicity of the Lebesgue Integral §integrable (with $a=b=1$) applied to the integrable functions $\psi_{1}g$ and $\psi_{2}g$ of Steps 4 and 5, the function $\phi\circ F$ is integrable with respect to $\gamma$ and

$$\int_{S}\phi\circ F\,d\gamma=H(\nu\,|\,\gamma)+\int_{S}\log g(T(s))\,\nu(ds).$$

Since $F$ is a density of $T_{\#}\nu$ with respect to $\gamma$ by Step 1 and $\phi\circ F$ is integrable with respect to $\gamma$, Relative Entropy of Probability Measures §relative-entropy shows that $T_{\#}\nu$ has finite relative entropy with respect to $\gamma$ and that $H(T_{\#}\nu\,|\,\gamma)=\int_{S}\phi\circ F\,d\gamma$, the value not depending on the choice of density as recorded in that definition. This is the asserted formula, and claim 2 is proved.