Relative Entropy over a Finite Partition: the Chain Rule, Images of the Pieces, Refinement and Approximation by Partition Entropies

In the settings of The Real Numbers: Standing Notation and Background and Measure Spaces and the Lebesgue Integral: Standing Notation, the measure space of the latter being instantiated at each use by the measure space named, let $(S,\mathcal{S})$ be a measurable space, let $\gamma$ and $\nu$ be probability measures on it, and suppose that $\nu$ has finite relative entropy $H(\nu\,|\,\gamma)$ with respect to $\gamma$. Finite measurable partitions and refinement are those of that definition, conditioned probability measures $\lambda(\cdot\,|\,A)$ are those of that definition, the partition entropy $h_{\mathcal{A}}(\nu\,|\,\gamma)$ is that of that definition, and image measures are those of claim 1 of Image Measures, Measures with Densities, and Change of Variables. In claims 1 and 2, $\mathcal{A}=(A_{1},\dots,A_{m})$ is a finite measurable partition of $S$, $p_{i}=\nu(A_{i})$ and $q_{i}=\gamma(A_{i})$ for $i\in[m]$, and the pieces of $\nu$ and of $\gamma$ on $A_{i}$ are the conditioned measures $\nu_{i}=\nu(\cdot\,|\,A_{i})$ whenever $0<p_{i}$, and $\gamma_{i}=\gamma(\cdot\,|\,A_{i})$ whenever $0<q_{i}$.

1. (Chain rule)¶ For every $i\in[m]$ with $0<p_{i}$ one has $0<q_{i}$, so that $h_{\mathcal{A}}(\nu\,|\,\gamma)$ is defined; for every such $i$ the piece $\nu_{i}$ has finite relative entropy with respect to the piece $\gamma_{i}$; and

$$H(\nu\,|\,\gamma)=h_{\mathcal{A}}(\nu\,|\,\gamma)+\sum_{i\in[m],\,0<p_{i}}p_{i}\,H(\nu_{i}\,|\,\gamma_{i}).$$

2. (Images of the pieces)¶ Let $(S',\mathcal{S}')$ be a measurable space and $R:S\to S'$ measurable with respect to $\mathcal{S}$ and $\mathcal{S}'$. Then for every $i\in[m]$ with $0<p_{i}$, so that $0<q_{i}$ by claim 1, the measure $R_{\#}\nu_{i}$ has finite relative entropy with respect to $R_{\#}\gamma_{i}$, and

$$h_{\mathcal{A}}(\nu\,|\,\gamma)+\sum_{i\in[m],\,0<p_{i}}p_{i}\,H(R_{\#}\nu_{i}\,|\,R_{\#}\gamma_{i})\le H(\nu\,|\,\gamma).$$

3. (Refinement)¶ Let $\mathcal{A}$ and $\mathcal{B}$ be finite measurable partitions of $S$ such that $\mathcal{B}$ refines $\mathcal{A}$; the partition entropies over $\mathcal{A}$ and over $\mathcal{B}$ are defined by claim 1. Then $h_{\mathcal{A}}(\nu\,|\,\gamma)\le h_{\mathcal{B}}(\nu\,|\,\gamma)\le H(\nu\,|\,\gamma)$.

4. (Approximation)¶ For every positive $\varepsilon\in\mathbb{R}$ there is a finite measurable partition $\mathcal{A}'$ of $S$ with $H(\nu\,|\,\gamma)-\varepsilon<h_{\mathcal{A}'}(\nu\,|\,\gamma)$.