The Partition Entropy of a Probability Measure Relative to Another over a Finite Measurable Partition

In the settings of The Real Numbers: Standing Notation and Background and Measure Spaces and the Lebesgue Integral: Standing Notation, the measure space of the latter being instantiated at each use by the measure space named, let $(S,\mathcal{S})$ be a measurable space, let $\gamma$ and $\nu$ be probability measures on it, and let $\mathcal{A}=(A_{1},\dots,A_{m})$ be a finite measurable partition of $S$. Write $p_{i}=\nu(A_{i})$ and $q_{i}=\gamma(A_{i})$ for $i\in[m]$; these lie in $[0,1]$ by monotonicity of $\nu$ and $\gamma$, since $A_{i}\subseteq S$ and $\nu(S)=\gamma(S)=1$. Let $P=\{i\in[m]:0<p_{i}\}$. By finite additivity of $\nu$ over the pairwise disjoint cells, $\sum_{i=1}^{m}p_{i}=\nu(S)=1$. Hence $P$ is nonempty, since otherwise every term would vanish and the sum would be $0$; and as the terms with $i\notin P$ vanish, $\sum_{i\in P}p_{i}=1$. $\log$ is the natural logarithm.

(Partition entropy)¶ Suppose that $0<q_{i}$ for every $i\in P$. The partition entropy of $\nu$ relative to $\gamma$ over $\mathcal{A}$ is the real number

$$h_{\mathcal{A}}(\nu\,|\,\gamma)=\sum_{i\in P}p_{i}\log\frac{p_{i}}{q_{i}},$$

each logarithm being applied to the positive real number $p_{i}/q_{i}$, and the sum being over the nonempty finite set $P$.