Proof of Variational Representation and Convexity of the Matrix Inverse

Throughout, $k$ and the notation are as in the statement; matrices are combined entrywise by the matrix sum, matrix difference, and scalar multiple. We first record the elementary componentwise identities used repeatedly. From Difference, Dot Product, and Orthogonality in $\mathbb{R}^n$, Matrix-Vector Product, and Euclidean Space $\mathbb{R}^n$, expanding the defining sums entrywise: the dot product is symmetric ($x\cdot y=\sum_ix_iy_i=y\cdot x$) and bilinear, and $Mx$ is linear in $x$ and satisfies $(\alpha M+\beta N)x=\alpha\,Mx+\beta\,Nx$ for matrices $M,N$ and reals $\alpha,\beta$. Next, for every symmetric matrix $M$ and all $x,y\in\mathbb{R}^{k}$,

using $M_{ij}=M_{ji}$. For matrices $A,B$ and $x\in\mathbb{R}^{k}$, with the matrix product $AB$,

so $A(Bx)=(AB)x$. Finally, for the identity matrix $I$ of Inverse Matrix and Invertible Real Square Matrix (diagonal entries $1$, off-diagonal entries $0$), $(Ix)_i=\sum_jI_{ij}x_j=x_i$, so $Ix=x$.

Step 1 (claim 1). By Invertibility of Symmetric Positive Definite Matrices, $M^{-1}$ exists and is symmetric positive definite. Fix $x,y\in\mathbb{R}^{k}$ and set $z=y-M^{-1}x$. By the identities above and the definition of the inverse, $M(M^{-1}x)=(MM^{-1})x=Ix=x$, so $Mz=My-x$ by linearity. Expanding by bilinearity,

By the symmetry identity applied to the symmetric matrix $M$, $(M^{-1}x)\cdot(My)=(M(M^{-1}x))\cdot y=x\cdot y$, and $(M^{-1}x)\cdot x=x\cdot(M^{-1}x)$ by symmetry of the dot product. Hence

Since $M$ is positive definite, $z\cdot(Mz)>0$ for $z\ne0$, while $z\cdot(Mz)=0$ for $z=0$ by bilinearity; so $z\cdot(Mz)\ge0$, with equality if and only if $z=0$, that is, if and only if $y=M^{-1}x$. Claim 1 follows.

Step 2 (claim 2). Write $C=\lambda A+(1-\lambda)B$. Entrywise, $C_{ij}=\lambda A_{ij}+(1-\lambda)B_{ij}=\lambda A_{ji}+(1-\lambda)B_{ji}=C_{ji}$, so $C$ is symmetric, and the identities above give, for every $x\in\mathbb{R}^{k}$,

For $x\ne0$ both $x\cdot(Ax)>0$ and $x\cdot(Bx)>0$, and $\lambda$, $1-\lambda$ are nonnegative with sum $1$, so at least one of them is positive and $x\cdot(Cx)>0$. Hence $C$ is positive definite and, by Invertibility of Symmetric Positive Definite Matrices, $C^{-1}$ exists and is symmetric positive definite.

Fix $x\in\mathbb{R}^{k}$ and put $y=C^{-1}x$. The equality case of claim 1 applied to $C$, together with the display above evaluated at this $y$, gives

where the splitting uses $\lambda+(1-\lambda)=1$ and the final bound is the inequality of claim 1 applied to $A$ and to $B$ (with the same $x$ and $y$). The matrix $D=\lambda A^{-1}+(1-\lambda)B^{-1}-C^{-1}$ is symmetric, being an entrywise combination of the symmetric matrices $A^{-1}$, $B^{-1}$, $C^{-1}$ (symmetric by Invertibility of Symmetric Positive Definite Matrices), and by bilinearity the display states exactly that $x\cdot(Dx)\ge0$ for every $x\in\mathbb{R}^{k}$. Hence $D$ is positive semidefinite, which is the stated inequality in the semidefinite order. $\blacksquare$