Proof of The Trace as a Sum of Quadratic Forms, its Monotonicity and a Norm Bound

Conventions. The notation is that of the statement; sums over an index range are the finite sums of $\mathbb{R}$, and $\le$ is the order of the ordered field of real numbers.

Claim 1. Fix $X\in\mathcal{S}(p)$.

Step 1. For every $k\in[m]$ and every $j\in[p]$,

$$(AX)_{kj}=(Xa_{k})_{j}.$$

Indeed, the definition of the product of real matrices gives $(AX)_{kj}=\sum_{l=1}^{p}A_{kl}X_{lj}$, while the definition of the matrix-vector product gives $(Xa_{k})_{j}=\sum_{l=1}^{p}X_{jl}(a_{k})_{l}$, and $(a_{k})_{l}=A_{kl}$ by the definition of the row $a_{k}$. Since $X$ is symmetric, $X_{jl}=X_{lj}$ by Euclidean Space and Real Matrices: Standing Notation (iii), and multiplication in $\mathbb{R}$ is commutative, so the two families of summands agree term by term and the two sums are equal.

Step 2. Both $A$ and $AX$ lie in $\mathcal{M}_{m\times p}(\mathbb{R})$, so claim 4 of Basic Properties of the Trace, applied with $A$ in the role of $U$ and $AX$ in the role of $V$, gives

$$\operatorname{tr}\bigl(A^{\top}(AX)\bigr)=\sum_{k=1}^{m}\ \sum_{j=1}^{p}A_{kj}(AX)_{kj}.$$

By Associativity of the Matrix Product the left-hand side is $\operatorname{tr}(A^{\top}AX)$. By Step 1 and the identity $(a_{k})_{j}=A_{kj}$, the inner sum equals $\sum_{j=1}^{p}(a_{k})_{j}(Xa_{k})_{j}$, which is $a_{k}\cdot(Xa_{k})$ by the coordinate formula for the dot product. This proves claim 1.

Claim 2. The identity matrix $I_{p}$ lies in $\mathcal{S}(p)$ by Euclidean Space and Real Matrices: Standing Notation (iii), so claim 1 applies with $X=I_{p}$. On the left, $A^{\top}AI_{p}=A^{\top}A$ by The Identity Matrix is a Two-Sided Multiplicative Identity. On the right, $I_{p}a_{k}=a_{k}$ by claim 2 of Linearity of the Matrix-Vector Product and the Quadratic Form as a Double Sum, and $a_{k}\cdot a_{k}=\lVert a_{k}\rVert^{2}$ by claim 1 of Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square. Claim 2 follows.

Claim 3. Let $X\in\mathcal{S}(p)$. The statement holds for every choice of the two dimensions and of the matrix, so we may apply claim 1 with $p$ in the role of $m$ and with $I_{p}$ in the role of $A$. By the definition of the identity matrix the entry $(I_{p})_{kj}$ equals $1$ if $k=j$ and $0$ otherwise, which is precisely the $j$th coordinate of $e_{k}$ by Euclidean Space and Real Matrices: Standing Notation (vi); hence the $k$th row of $I_{p}$ is $e_{k}$. Moreover $I_{p}$ is symmetric by Euclidean Space and Real Matrices: Standing Notation (iii), so $I_{p}^{\top}=I_{p}$, and $I_{p}I_{p}X=I_{p}X=X$ by The Identity Matrix is a Two-Sided Multiplicative Identity. Claim 1 therefore reads $\operatorname{tr}(X)=\sum_{k=1}^{p}e_{k}\cdot(Xe_{k})$.

Claim 4. Let $X,Y\in\mathcal{S}(p)$ satisfy $X\preceq Y$. By Euclidean Space and Real Matrices: Standing Notation (iv) we have $z\cdot(Xz)\le z\cdot(Yz)$ for every $z\in\mathbb{R}^{p}$, and in particular $a_{k}\cdot(Xa_{k})\le a_{k}\cdot(Ya_{k})$ for every $k\in[m]$. Claim 1 of Comparison and Absolute Value Bounds for Finite Sums of Real Numbers gives

$$\sum_{k=1}^{m}a_{k}\cdot(Xa_{k})\le\sum_{k=1}^{m}a_{k}\cdot(Ya_{k}),$$

and claim 1 above identifies the two sides with $\operatorname{tr}(A^{\top}AX)$ and $\operatorname{tr}(A^{\top}AY)$. Running the same argument with $e_{k}$ in place of $a_{k}$ and invoking claim 3 in place of claim 1 gives $\operatorname{tr}(X)\le\operatorname{tr}(Y)$.

Claim 5. Let $X\in\mathcal{S}(p)$. By claim 3 above and by claim 2 of Comparison and Absolute Value Bounds for Finite Sums of Real Numbers,

$$\bigl|\operatorname{tr}(X)\bigr|\le\sum_{k=1}^{p}\bigl|e_{k}\cdot(Xe_{k})\bigr| .$$

For each $k\in[p]$, claim 2 of Properties of the Norm of a Symmetric Real Matrix gives $|e_{k}\cdot(Xe_{k})|\le\lVert X\rVert\lVert e_{k}\rVert^{2}$, and $\lVert e_{k}\rVert=1$ by Euclidean Space and Real Matrices: Standing Notation (vi), so $\lVert e_{k}\rVert^{2}=1$ and hence $|e_{k}\cdot(Xe_{k})|\le\lVert X\rVert$. Therefore, by claim 1 of Comparison and Absolute Value Bounds for Finite Sums of Real Numbers and claim 3 of Properties of Finite Sums,

$$\sum_{k=1}^{p}\bigl|e_{k}\cdot(Xe_{k})\bigr|\le\sum_{k=1}^{p}\lVert X\rVert=\lVert X\rVert\sum_{k=1}^{p}1=\beta_{p}\lVert X\rVert ,$$

the last two equalities using $\lVert X\rVert\cdot1=\lVert X\rVert$ and the commutativity of multiplication. Transitivity of $\le$ now gives claim 5.