Relative Entropy of Probability Measures

In the setting of Measure Spaces and the Lebesgue Integral: Standing Notation, let $(X,\mathcal{F})$ be a measurable space, let $\nu$ and $\gamma$ be probability measures on it, let $\phi$ be the function $s\mapsto s\log s$ of The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm, and let densities of $\nu$ with respect to $\gamma$ be those of The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities. For such a density $f$ the function $\phi\circ f$ is measurable by The Function $s\log s$: Continuity, Young's Inequality and Lower Bounds, with the Elementary Bounds for the Exponential and the Logarithm §continuous.

(Relative entropy)¶ The measure $\nu$ has finite relative entropy with respect to $\gamma$ if it has a density $f$ with respect to $\gamma$ for which $\phi\circ f$ is integrable with respect to $\gamma$. The relative entropy of such a $\nu$ with respect to $\gamma$ is the real number

It does not depend on the choice of $f$: two densities of $\nu$ with respect to $\gamma$ agree outside a set of $\gamma$-measure $0$ by The Radon-Nikodym Theorem for a Finite Measure and a Sigma-Finite Measure, and Uniqueness of Densities §uniqueness, hence so do the two functions $\phi\circ f$, and their integrals agree by The Lebesgue Integral and Null Sets: Almost-Everywhere Comparison, Markov's Inequality, and Dominated Convergence Almost Everywhere §comparison.