Proof of Equality of Mixed Partial Derivatives, and Symmetry and Continuity of the Hessian

Throughout, the arithmetic and order of $\mathbb{R}$ are used through The Structure of ℝ as a Field and an Ordered Ring (i), The Structure of ℝ as a Field and an Ordered Ring (ii), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (v), Arithmetic and Order Rules in the Real Numbers, Stated for ℝ (ix), Rules of Arithmetic and Order in an Ordered Field (iii) and The Quotient 1/x, Order Rules for Sums, and Nesting of Closed Intervals in ℝ (i); distances in $\mathbb{R}^{n}$ through Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (ii); and the metric spaces $(\mathbb{R}^{n},d_{E})$ and $(\mathbb{R},d_{\mathbb{R}})$ through $\mathbb{R}^n$ with the Euclidean Distance Is a Complete Metric Space with Coordinatewise Convergence (i) and The Absolute Value Metric on the Real Numbers (i). The argument for (i) is that of Rudin's Theorem 9.41, specialised to functions of class $C^{2}$.

We first collect the regularity of $f$ that the argument uses. Since $f$ is of class $C^{2}$, it is of class $C^{1}$ by Basic Properties of Cᵏ Functions (ii), hence differentiable on $U$ by Basic Properties of Cᵏ Functions (v). For every $i\in[n]$, $\partial_{i}f$ is of class $C^{1}$ on $U$ by Basic Properties of Cᵏ Functions (iii), hence also differentiable on $U$. Applying Basic Properties of Cᵏ Functions (iii) once more, every $\partial_{i}\partial_{j}f$ with $i,j\in[n]$ is of class $C^{0}$ on $U$, hence continuous on $U$ by Basic Properties of Cᵏ Functions (iv). Finally, let $h:U\to\mathbb{R}$ be differentiable at a point $z\in U$, so that its gradient $\nabla h(z)$ exists. Since the $i$-th coordinate of a vector equals its dot product with the standard basis vector $e_{i}$ (Standard Basis Vectors of ℝ^q: Coordinates, Orthonormality and Unit Length (i) and The Dot Product on $\mathbb{R}^n$ Is Symmetric, Bilinear and Positive Definite (i)), and the $i$-th coordinate of $\nabla h(z)$ is $\partial_{i}h(z)$ (Gradient of a Real-Valued Function on an Open Subset of $\mathbb{R}^n$ (i)),

$$\nabla h(z)\cdot e_{i}=\partial_{i}h(z)\qquad(i\in[n]).$$

In particular this holds at every $z\in U$ for $h=f$ and for $h=\partial_{l}f$ with $l\in[n]$, both being differentiable on $U$.

We now prove (i). Fix $x\in U$ and $i,j\in[n]$. For $i=j$ there is nothing to prove, so let $i\neq j$. Since $U$ is open, there is $\rho>0$ with $B(x,\rho)\subseteq U$. Let $s,t$ be real numbers with $|s|,|t|<\rho/2$, and let $\sigma,\tau$ be real numbers with $|\sigma|\le|s|$ and $|\tau|\le|t|$. The standard basis vectors have norm one (Standard Basis Vectors of ℝ^q: Coordinates, Orthonormality and Unit Length (iii)), so by the triangle inequality and homogeneity of the norm (Properties of the Euclidean Norm on $\mathbb{R}^n$: Square, Distance, Vanishing, Coordinate Bounds, Homogeneity, Triangle Inequalities and Expansion of a Square (v))

$$\lVert(x+\sigma e_{i}+\tau e_{j})-x\rVert=\lVert\sigma e_{i}+\tau e_{j}\rVert\le|\sigma|+|\tau|<\rho ,$$

and all such points $x+\sigma e_{i}+\tau e_{j}$ lie in $B(x,\rho)\subseteq U$. In particular, when $s=t>0$, each of these points lies within distance $|\sigma|+|\tau|\le2s$ of $x$. The proof compares two evaluations of the second difference

$$\Delta(s,t)=f(x+se_{i}+te_{j})-f(x+se_{i})-f(x+te_{j})+f(x).$$

For the first evaluation, let $0<|s|,|t|<\rho/2$, and consider on the interval $(-\rho/2,\rho/2)$ the function

$$g(\sigma)=f(x+te_{j}+\sigma e_{i})-f(x+\sigma e_{i}).$$

Both terms are values of $f$ along lines in direction $e_{i}$ that stay in $U$, one through the base point $x+te_{j}$ and one through the base point $x$. By Derivatives of a Function Along a Line Segment in ℝⁿ (ii), applied to $f$ with direction $e_{i}$ and each of these base points, together with the rule for the derivative of a difference, a sum with $-1$ times the second term (Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (iv) and Derivatives of Constants, the Identity, Sums, Scalar Multiples, Products, Reciprocals, Quotients and Powers (v)), $g$ is differentiable with

$$g'(\sigma)=\partial_{i}f(x+\sigma e_{i}+te_{j})-\partial_{i}f(x+\sigma e_{i}).$$

Since $\Delta(s,t)=g(s)-g(0)$, and $g$ is continuous on $(-\rho/2,\rho/2)$, being differentiable there (The Increment Form of the Derivative, and Differentiability Implies Continuity (ii)), the mean value theorem (on $[0,s]$ or $[s,0]$) gives a number $\sigma$ strictly between $0$ and $s$ with $\Delta(s,t)=s\,g'(\sigma)$. With this $\sigma$ fixed, consider on $(-\rho/2,\rho/2)$ the function $k(\tau)=\partial_{i}f(x+\sigma e_{i}+\tau e_{j})$. Applying Derivatives of a Function Along a Line Segment in ℝⁿ (ii) to the differentiable function $\partial_{i}f$, with base point $x+\sigma e_{i}$ and direction $e_{j}$, shows that $k$ is differentiable with

$$k'(\tau)=\partial_{j}\partial_{i}f(x+\sigma e_{i}+\tau e_{j}),$$

a second partial derivative of $f$. As $g'(\sigma)=k(t)-k(0)$, and $k$ is continuous on $(-\rho/2,\rho/2)$ by The Increment Form of the Derivative, and Differentiability Implies Continuity (ii), the mean value theorem (on $[0,t]$ or $[t,0]$) gives a number $\tau$ strictly between $0$ and $t$ with $g'(\sigma)=t\,k'(\tau)$. Combining the two steps,

$$\Delta(s,t)=st\,\partial_{j}\partial_{i}f(p),\qquad p=x+\sigma e_{i}+\tau e_{j}.$$

For the second evaluation, observe that $\Delta(s,t)$ is unchanged when the roles of $(i,s)$ and $(j,t)$ are exchanged: the first and last terms do not change, and the two middle terms are interchanged. The first evaluation with $i$ and $j$ exchanged (after renaming the intermediate parameters so that $\sigma'$ is the coefficient of $e_{i}$ and $\tau'$ that of $e_{j}$) therefore gives $\sigma'$ strictly between $0$ and $s$ and $\tau'$ strictly between $0$ and $t$ with

$$\Delta(s,t)=st\,\partial_{i}\partial_{j}f(q),\qquad q=x+\sigma' e_{i}+\tau' e_{j}.$$

It remains to let $s$ and $t$ tend to $0$. Let $\varepsilon>0$. First, by continuity of $\partial_{j}\partial_{i}f$ and $\partial_{i}\partial_{j}f$ at $x$ (Continuous Map Between Metric Spaces (i)), choose $\delta\in(0,\rho]$ such that every $z\in B(x,\delta)$ satisfies

$$|\partial_{j}\partial_{i}f(z)-\partial_{j}\partial_{i}f(x)|<\varepsilon\quad\text{and}\quad|\partial_{i}\partial_{j}f(z)-\partial_{i}\partial_{j}f(x)|<\varepsilon .$$

Then, for this $\delta$, take $s=t$ with $0<s<\delta/2$. The points $p$ and $q$ lie within distance $2s<\delta$ of $x$, as shown above, so they lie in $B(x,\delta)$. Dividing the two evaluations of $\Delta(s,s)$ by $s^{2}$ gives $\partial_{j}\partial_{i}f(p)=\partial_{i}\partial_{j}f(q)$, and therefore

$$|\partial_{j}\partial_{i}f(x)-\partial_{i}\partial_{j}f(x)|\le|\partial_{j}\partial_{i}f(x)-\partial_{j}\partial_{i}f(p)|+|\partial_{i}\partial_{j}f(q)-\partial_{i}\partial_{j}f(x)|<2\varepsilon .$$

Since $\varepsilon>0$ was arbitrary, $\partial_{i}\partial_{j}f(x)=\partial_{j}\partial_{i}f(x)$.

For (ii), let $x\in U$. By The Hessian Matrix of a Function of Class C² (i) and (i),

$$\bigl(D^{2}f(x)\bigr)_{ij}=\partial_{i}\partial_{j}f(x)=\partial_{j}\partial_{i}f(x)=\bigl(D^{2}f(x)\bigr)_{ji}\qquad\text{for all }i,j\in[n],$$

and a matrix $A$ is symmetric exactly when $A_{ij}=A_{ji}$ for all $i,j$ (Symmetric Real Matrices and the Set $\mathcal{S}(n)$ (iii)); so $D^{2}f(x)\in\mathcal{S}(n)$.

For (iii), the entry functions $x\mapsto\bigl(D^{2}f(x)\bigr)_{ij}$ of the Hessian function are the functions $\partial_{i}\partial_{j}f$, which are continuous on $U$, as noted at the start. Hence $D^{2}f$ is continuous on $U$ by Continuity of Vector- and Matrix-Valued Maps: the Componentwise Criterion, Norms and Algebraic Operations (ii) and Continuity of Vector- and Matrix-Valued Maps: the Componentwise Criterion, Norms and Algebraic Operations (v).