Measures on a Finite Set Given by Point Masses: the Measure, Integrals as Finite Sums, and Relative Entropy

In the setting of The Real Numbers: Standing Notation and Background, let $Y$ be a nonempty finite set, and let $\mathcal{P}(Y)$ be the power set of $Y$, a $\sigma$-algebra on $Y$. Sums over finite index sets are those of Sum over a Finite Index Set. For a map $p:Y\to\mathbb{R}$ with $0\le p(s)$ for every $s\in Y$ and for $A\subseteq Y$ put $\lambda_{p}(A)=\sum_{s\in A}p(s)$ if $A\neq\emptyset$, a sum over the set $A$, which is finite by claim 3 of Basic Properties of Finite Sets, and $\lambda_{p}(\emptyset)=0$; $\lambda_{p}$ is the measure with point masses $p$. Measures and probability measures are those of Measure, Measure Space, and Probability Measure; measurability, integrals and integrability are those of Measure Spaces and the Lebesgue Integral: Standing Notation, read with the measure space named in each claim. Finite relative entropy and $H(\cdot\,|\,\cdot)$ are those of Relative Entropy of Probability Measures §relative-entropy, and densities are those of The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities. $\phi:[0,\infty)\to\mathbb{R}$ is the function of The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm, with $\phi(0)=0$ and $\phi(r)=r\log r$ for positive $r$, $\log$ being the natural logarithm.

1. (Measure)¶ For every map $p:Y\to\mathbb{R}$ with $0\le p(s)$ for every $s\in Y$, $\lambda_{p}$ is a measure on $(Y,\mathcal{P}(Y))$ with $\lambda_{p}(\{s\})=p(s)$ for every $s\in Y$; it is a probability measure if and only if $\sum_{s\in Y}p(s)=1$. Conversely, every probability measure $\lambda$ on $(Y,\mathcal{P}(Y))$ equals $\lambda_{p}$ for the map $p(s)=\lambda(\{s\})$ (a real number, since $\lambda(\{s\})\le\lambda(Y)=1$).

2. (Integrals)¶ For every map $p:Y\to\mathbb{R}$ with $0\le p(s)$ for every $s\in Y$, every map $f:Y\to\mathbb{R}$ is measurable with respect to $\mathcal{P}(Y)$ and integrable with respect to $\lambda_{p}$, and

$$\int_{Y}f\,d\lambda_{p}=\sum_{s\in Y}f(s)\,p(s).$$

3. (Relative entropy)¶ Let $p,p':Y\to\mathbb{R}$ satisfy $0\le p(s)$ and $0<p'(s)$ for every $s\in Y$ and $\sum_{s\in Y}p(s)=\sum_{s\in Y}p'(s)=1$, so that $\lambda_{p}$ and $\lambda_{p'}$ are probability measures by claim 1. Then the map $s\mapsto p(s)/p'(s)$ is a density of $\lambda_{p}$ with respect to $\lambda_{p'}$, $\lambda_{p}$ has finite relative entropy with respect to $\lambda_{p'}$, and

$$H(\lambda_{p}\,|\,\lambda_{p'})=\sum_{s\in Y}\phi\Bigl(\frac{p(s)}{p'(s)}\Bigr)\,p'(s).$$