Rules for Expectations, Variances and Covariances

In the setting of Probability Spaces: Notation, let $m,n,r,s\in\mathbb{N}$.

(i) Events. For every event $B$,

$$\mathbb{E}\,\mathbf{1}_{B}=P(B).$$

(ii) Constants. For every $c\in\mathbb{R}$, the constant random variable $c$ is integrable and $\mathbb{E}\,c=c$.

(iii) Linearity. Let $X,Y\in\mathcal{L}^{1}(P)$ and $a,b\in\mathbb{R}$. Then $aX+bY\in\mathcal{L}^{1}(P)$ and

$$\mathbb{E}(aX+bY)=a\,\mathbb{E}X+b\,\mathbb{E}Y.$$

(iv) Monotonicity. Let $X,Y\in\mathcal{L}^{1}(P)$ with $X\le Y$. Then $\mathbb{E}X\le\mathbb{E}Y$.

(v) Absolute value. Let $X\in\mathcal{L}^{1}(P)$. Then $|X|\in\mathcal{L}^{1}(P)$ and

$$|\mathbb{E}X|\le\mathbb{E}|X|.$$

(vi) Affine functions of random vectors. Let $X$ be an integrable random vector in $\mathbb{R}^{n}$, $A\in\mathbb{R}^{m\times n}$ and $b\in\mathbb{R}^{m}$. Then $AX+b$ is integrable and

$$\mathbb{E}(AX+b)=A\,\mathbb{E}X+b.$$

(vii) Covariance and variance formulas. Let $X,Y\in\mathcal{L}^{2}(P)$. Then $XY\in\mathcal{L}^{1}(P)$ and

$$\operatorname{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}X\,\mathbb{E}Y,\qquad\operatorname{Var}X=\mathbb{E}[X^{2}]-(\mathbb{E}X)^{2}.$$

(viii) Variances are nonnegative. Let $X\in\mathcal{L}^{2}(P)$. Then $\operatorname{Var}X\ge0$.

(ix) Variance of an affine function. Let $X\in\mathcal{L}^{2}(P)$ and $a,b\in\mathbb{R}$. Then $aX+b\in\mathcal{L}^{2}(P)$ and

$$\operatorname{Var}(aX+b)=a^{2}\operatorname{Var}X.$$

(x) Covariance is symmetric and bilinear. Let $X,Y,Z\in\mathcal{L}^{2}(P)$ and $a,b,c\in\mathbb{R}$. Then, for the constant random variable $c$,

$$\operatorname{Cov}(X,Y)=\operatorname{Cov}(Y,X),\qquad\operatorname{Cov}(aX+bY,Z)=a\operatorname{Cov}(X,Z)+b\operatorname{Cov}(Y,Z),\qquad\operatorname{Cov}(X,c)=0.$$

(xi) Covariance matrices as expectations. Let $X\in\mathcal{L}^{2}(P;\mathbb{R}^{n})$ and $Y\in\mathcal{L}^{2}(P;\mathbb{R}^{r})$. Then $X$ and $Y$ are integrable, $(X-\mathbb{E}X)(Y-\mathbb{E}Y)^{\top}$ is integrable, and

$$\operatorname{Cov}(X,Y)=\mathbb{E}\bigl[(X-\mathbb{E}X)(Y-\mathbb{E}Y)^{\top}\bigr],\qquad\operatorname{Cov}(X)=\operatorname{Cov}(X,X).$$

(xii) Covariance matrices are positive semidefinite. Let $X\in\mathcal{L}^{2}(P;\mathbb{R}^{n})$. Then $\operatorname{Cov}(X)$ is symmetric and positive semidefinite.

(xiii) Covariance of affine functions. Let $X\in\mathcal{L}^{2}(P;\mathbb{R}^{n})$, $Y\in\mathcal{L}^{2}(P;\mathbb{R}^{r})$, $A\in\mathbb{R}^{m\times n}$, $b\in\mathbb{R}^{m}$, $C\in\mathbb{R}^{s\times r}$ and $d\in\mathbb{R}^{s}$. Then $AX+b\in\mathcal{L}^{2}(P;\mathbb{R}^{m})$, $CY+d\in\mathcal{L}^{2}(P;\mathbb{R}^{s})$, and

$$\operatorname{Cov}(AX+b,\,CY+d)=A\operatorname{Cov}(X,Y)\,C^{\top},\qquad\operatorname{Cov}(AX+b)=A\operatorname{Cov}(X)\,A^{\top}.$$

(xiv) Second-moment matrix. Let $X\in\mathcal{L}^{2}(P;\mathbb{R}^{n})$. Then $XX^{\top}$ is integrable and

$$\operatorname{Cov}(X)=\mathbb{E}[XX^{\top}]-(\mathbb{E}X)(\mathbb{E}X)^{\top}.$$