Proof of Rules for Expectations, Variances and Covariances

Throughout, expectations are integrals against $P$ (Expectation (ii), Expectation (iii), Expectation (iv)); constant functions and pointwise sums, products and matrix products of measurable functions are measurable (Measurable Maps: Generators, Composition, Continuous Maps, Restrictions, and Criteria for Real- and [0,∞]-Valued Maps (iii), Arithmetic of Measurable Real- and [0,∞]-Valued Maps (i), Sums, Products, Transposes, Dot Products and Norms of Measurable Vector- and Matrix-Valued Maps (i), Sums, Products, Transposes, Dot Products and Norms of Measurable Vector- and Matrix-Valued Maps (ii), Vector- and Matrix-Valued Measurable Maps: Components, Entries and Continuous Images (ii)); and for $X,Y\in\mathcal{L}^{2}(P)$ we have $X,Y\in\mathcal{L}^{1}(P)$, constants lie in $\mathcal{L}^{2}(P)$, the centred functions $X-\mathbb{E}X$, $Y-\mathbb{E}Y$ lie in $\mathcal{L}^{2}(P)$, and products of elements of $\mathcal{L}^{2}(P)$ lie in $\mathcal{L}^{1}(P)$ (Lᵖ Inclusions and Constants on a Finite Measure Space (i), Elementary Properties of the p-Seminorm (i), Lᵖ Inclusions and Constants on a Finite Measure Space (ii), Minkowski's Inequality (i), Hölder's Inequality (ii)).

Events. By Basic Properties of the Integral of Nonnegative Measurable Maps (ii), $\mathbf{1}_{B}\in\mathcal{M}^{+}(\Omega,\mathcal{F})$ and $\int_{\Omega}\mathbf{1}_{B}\,dP=P(B)$. Since $P(B)\le P(\Omega)=1$ (Basic Properties of a Measure (iii)), the real-valued function $\mathbf{1}_{B}$ is also integrable, with the same integral (Measurable Maps: Generators, Composition, Continuous Maps, Restrictions, and Criteria for Real- and [0,∞]-Valued Maps (vii), Linearity and Monotonicity of the Lebesgue Integral, and the Integrability Criterion (vi)). So $\mathbb{E}\,\mathbf{1}_{B}=P(B)$ for either reading of the expectation.

Constants. By Lᵖ Inclusions and Constants on a Finite Measure Space (ii) with exponent $1$, the constant $c$ is integrable and

$$\mathbb{E}\,c=c\,P(\Omega)=c.$$

Linearity, monotonicity and absolute value. These are Linearity and Monotonicity of the Lebesgue Integral, and the Integrability Criterion (vii), Linearity and Monotonicity of the Lebesgue Integral, and the Integrability Criterion (ix) and Linearity and Monotonicity of the Lebesgue Integral, and the Integrability Criterion (x) for $\mu=P$.

Affine functions of random vectors. By Rules for Integrals of Vector- and Matrix-Valued Functions (v), $AX$ is integrable and $\mathbb{E}(AX)=A\,\mathbb{E}X$. The constant function $b$ has the constant components $b_{k}$, so by (ii) it is integrable with $\mathbb{E}\,b=b$ (Integrable Vector- and Matrix-Valued Functions and Their Integrals (i), Integrable Vector- and Matrix-Valued Functions and Their Integrals (ii)). By Rules for Integrals of Vector- and Matrix-Valued Functions (iii) with both coefficients $1$, $AX+b$ is integrable and $\mathbb{E}(AX+b)=A\,\mathbb{E}X+b$.

Covariance and variance formulas. Pointwise,

$$(X-\mathbb{E}X)(Y-\mathbb{E}Y)=XY-(\mathbb{E}Y)X-(\mathbb{E}X)Y+\mathbb{E}X\,\mathbb{E}Y,$$

so (iii) and (ii) give

$$\operatorname{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}Y\,\mathbb{E}X-\mathbb{E}X\,\mathbb{E}Y+\mathbb{E}X\,\mathbb{E}Y=\mathbb{E}[XY]-\mathbb{E}X\,\mathbb{E}Y$$

(Variance and Covariance (i)). For $Y=X$ this is the formula for $\operatorname{Var}X=\operatorname{Cov}(X,X)$ (Variance and Covariance (ii)).

Variances are nonnegative. Since $(X-\mathbb{E}X)^{2}\ge0$ pointwise, (iv) and (ii) give

$$\operatorname{Var}X=\mathbb{E}\bigl[(X-\mathbb{E}X)^{2}\bigr]\ge\mathbb{E}\,0=0$$

(Variance and Covariance (i), Variance and Covariance (ii)).

Variance of an affine function. The constant $b$ lies in $\mathcal{L}^{2}(P)$ (Lᵖ Inclusions and Constants on a Finite Measure Space (ii)), so $aX+b\in\mathcal{L}^{2}(P)$ (Minkowski's Inequality (i)). By (iii) and (ii), $\mathbb{E}(aX+b)=a\,\mathbb{E}X+b$, so $(aX+b)-\mathbb{E}(aX+b)=a(X-\mathbb{E}X)$, and by Variance and Covariance (i), Variance and Covariance (ii) and (iii),

$$\operatorname{Var}(aX+b)=\mathbb{E}\bigl[a^{2}(X-\mathbb{E}X)^{2}\bigr]=a^{2}\,\mathbb{E}\bigl[(X-\mathbb{E}X)^{2}\bigr]=a^{2}\operatorname{Var}X .$$

Covariance is symmetric and bilinear. Symmetry holds because multiplication in $\mathbb{R}$ is commutative (Variance and Covariance (i)). Next, $aX+bY\in\mathcal{L}^{2}(P)$ (Minkowski's Inequality (i)) and $\mathbb{E}(aX+bY)=a\,\mathbb{E}X+b\,\mathbb{E}Y$, so

$$\bigl(aX+bY-\mathbb{E}(aX+bY)\bigr)(Z-\mathbb{E}Z)=a(X-\mathbb{E}X)(Z-\mathbb{E}Z)+b(Y-\mathbb{E}Y)(Z-\mathbb{E}Z),$$

and (iii) gives $\operatorname{Cov}(aX+bY,Z)=a\operatorname{Cov}(X,Z)+b\operatorname{Cov}(Y,Z)$. Finally the constant $c$ lies in $\mathcal{L}^{2}(P)$ (Lᵖ Inclusions and Constants on a Finite Measure Space (ii)) and $\mathbb{E}\,c=c$, so $(X-\mathbb{E}X)(c-\mathbb{E}\,c)=0$ and $\operatorname{Cov}(X,c)=\mathbb{E}\,0=0$.

Covariance matrices as expectations. The components of $X$ and $Y$ lie in $\mathcal{L}^{2}(P)\subseteq\mathcal{L}^{1}(P)$ (Square-Integrable Vector-Valued Functions: Components, Dot Products and Almost-Everywhere Equality (i)), so $X$ and $Y$ are integrable, with $(\mathbb{E}X)_{k}=\mathbb{E}X_{k}$ (Integrable Vector- and Matrix-Valued Functions and Their Integrals (i), Expectation (iii)). Put $Z=X-\mathbb{E}X$ and $W=Y-\mathbb{E}Y$, with components $Z_{k}=X_{k}-\mathbb{E}X_{k}$ and $W_{l}=Y_{l}-\mathbb{E}Y_{l}$ in $\mathcal{L}^{2}(P)$. The entries $Z_{k}W_{l}$ of $ZW^{\top}$ lie in $\mathcal{L}^{1}(P)$, so $ZW^{\top}$ is integrable (Integrable Vector- and Matrix-Valued Functions and Their Integrals (iii)), and for all $k\in[n]$, $l\in[r]$, by Expectation (iv), Variance and Covariance (i) and Variance and Covariance (v),

$$\bigl(\mathbb{E}[ZW^{\top}]\bigr)_{kl}=\mathbb{E}[Z_{k}W_{l}]=\operatorname{Cov}(X_{k},Y_{l})=\operatorname{Cov}(X,Y)_{kl}.$$

So $\operatorname{Cov}(X,Y)=\mathbb{E}[ZW^{\top}]$. Moreover $\operatorname{Cov}(X)=\operatorname{Cov}(X,X)$, both matrices having the entries $\operatorname{Cov}(X_{i},X_{j})$ (Variance and Covariance (iv), Variance and Covariance (v)).

Covariance matrices are positive semidefinite. By (x), $\operatorname{Cov}(X)_{ij}=\operatorname{Cov}(X_{i},X_{j})=\operatorname{Cov}(X_{j},X_{i})=\operatorname{Cov}(X)_{ji}$, so $\operatorname{Cov}(X)$ is symmetric (Symmetric Real Matrices and the Set $\mathcal{S}(n)$ (iii)). Let $u\in\mathbb{R}^{n}$ and $Z=X-\mathbb{E}X$. By The Dot Product on $\mathbb{R}^n$ (i) and Matrix-Vector Product (i), then Variance and Covariance (i), then Linearity and Monotonicity of the Lebesgue Integral, and the Integrability Criterion (viii) (for each $i$, and then for the sum over $i$),

$$u\cdot\bigl(\operatorname{Cov}(X)u\bigr)=\sum_{i=1}^{n}u_{i}\sum_{j=1}^{n}\operatorname{Cov}(X_{i},X_{j})\,u_{j}=\sum_{i=1}^{n}u_{i}\sum_{j=1}^{n}u_{j}\,\mathbb{E}[Z_{i}Z_{j}]=\mathbb{E}\Bigl[\sum_{i=1}^{n}u_{i}\sum_{j=1}^{n}u_{j}Z_{i}Z_{j}\Bigr]=\mathbb{E}\bigl[(u\cdot Z)^{2}\bigr]\ge0,$$

the inequality by (iv) and (ii). So $\operatorname{Cov}(X)$ is positive semidefinite (Positive Semidefinite and Positive Definite Real Matrices (i)).

Covariance of affine functions. First, $AX+b\in\mathcal{L}^{2}(P;\mathbb{R}^{m})$. With $c_{A}=\sum_{i=1}^{m}\sum_{j=1}^{n}A_{ij}^{2}$ we have $\lVert AX(\omega)\rVert^{2}\le c_{A}\lVert X(\omega)\rVert^{2}$ for every $\omega\in\Omega$ (Algebraic Rules for Matrix Products, Matrix-Vector Products and Transposes (ix)), that is, $|AX|^{2}\le c_{A}|X|^{2}$ for the functions of Square-Integrable Vector-Valued Functions (i); so

$$\int_{\Omega}|AX|^{2}\,dP\le c_{A}\int_{\Omega}|X|^{2}\,dP<\infty$$

(Basic Properties of the Integral of Nonnegative Measurable Maps (iii), Basic Properties of the Integral of Nonnegative Measurable Maps (iv), Square-Integrable Vector-Valued Functions (ii)). The constant $b$ has constant components, so it lies in $\mathcal{L}^{2}(P;\mathbb{R}^{m})$ (Lᵖ Inclusions and Constants on a Finite Measure Space (ii), Square-Integrable Vector-Valued Functions: Components, Dot Products and Almost-Everywhere Equality (i)), and $AX+b\in\mathcal{L}^{2}(P;\mathbb{R}^{m})$ by Square-Integrable Vector-Valued Functions: Components, Dot Products and Almost-Everywhere Equality (iii). Likewise $CY+d\in\mathcal{L}^{2}(P;\mathbb{R}^{s})$.

Let $Z=X-\mathbb{E}X$ and $W=Y-\mathbb{E}Y$. By (vi) and Algebraic Rules for Matrix Products, Matrix-Vector Products and Transposes (v), the centred functions are

$$(AX+b)-\mathbb{E}(AX+b)=AZ,\qquad(CY+d)-\mathbb{E}(CY+d)=CW .$$

For $i\in[m]$ and $j\in[s]$, by Matrix-Vector Product (i), Product of Real Matrices (i) and Transpose of a Real Matrix (i),

$$\bigl((AZ)(CW)^{\top}\bigr)_{ij}=\Bigl(\sum_{k=1}^{n}A_{ik}Z_{k}\Bigr)\Bigl(\sum_{l=1}^{r}C_{jl}W_{l}\Bigr)=\sum_{l=1}^{r}\Bigl(\sum_{k=1}^{n}A_{ik}Z_{k}W_{l}\Bigr)C_{jl}=\bigl((A\,ZW^{\top})C^{\top}\bigr)_{ij}.$$

So, by (xi) for $AX+b$ and $CY+d$, then Rules for Integrals of Vector- and Matrix-Valued Functions (vi), then (xi) for $X$ and $Y$,

$$\operatorname{Cov}(AX+b,CY+d)=\mathbb{E}\bigl[(AZ)(CW)^{\top}\bigr]=\mathbb{E}\bigl[A\,ZW^{\top}C^{\top}\bigr]=A\,\mathbb{E}[ZW^{\top}]\,C^{\top}=A\operatorname{Cov}(X,Y)\,C^{\top}.$$

Taking $Y=X$, $C=A$ and $d=b$, and using $\operatorname{Cov}(X)=\operatorname{Cov}(X,X)$ and $\operatorname{Cov}(AX+b)=\operatorname{Cov}(AX+b,AX+b)$ ((xi)), gives $\operatorname{Cov}(AX+b)=A\operatorname{Cov}(X)A^{\top}$.

Second-moment matrix. The components $X_{i}$ lie in $\mathcal{L}^{2}(P)$ (Square-Integrable Vector-Valued Functions: Components, Dot Products and Almost-Everywhere Equality (i)), so the entries $X_{i}X_{j}$ of $XX^{\top}$ lie in $\mathcal{L}^{1}(P)$ and $XX^{\top}$ is integrable (Integrable Vector- and Matrix-Valued Functions and Their Integrals (iii)). For $i,j\in[n]$, by Variance and Covariance (iv), (vii), Expectation (iv) and Expectation (iii),

$$\operatorname{Cov}(X)_{ij}=\operatorname{Cov}(X_{i},X_{j})=\mathbb{E}[X_{i}X_{j}]-\mathbb{E}X_{i}\,\mathbb{E}X_{j}=\bigl(\mathbb{E}[XX^{\top}]\bigr)_{ij}-\bigl((\mathbb{E}X)(\mathbb{E}X)^{\top}\bigr)_{ij},$$

and matrices with the same entries are equal.