Proof of The Limiting Cost Along the Approximate Kalman Policy Is the Optimal Value of the Fluctuation LQG Problem

Throughout, (H1)--(H4) and conclusion 1'', conclusion 2'' refer to the approximate Kalman filter and policy lemma, whose setting and hypotheses are in force; the linear-Gaussian state-observation model, its data $(A^\circ,\varepsilon^\circ,\tilde{E}^\circ,\tilde{\varepsilon}^\circ,\xi,W^\circ)$, the control dimension $m$, the control matrix assignment $B(t)=\mathcal{B}_t$, and the assignments $Q^\circ(t)=Q_t$, $V^\circ(t)=\tfrac12V_t$, $R^\circ(t)=R_t$, $F^\circ=\hat{F}$ are those of the statement. Products of matrices are matrix products, $(\cdot)^{\top}$ is the transpose, and integrals of matrix-valued maps are entrywise Riemann integrals of continuous integrands, existing by continuity, unless a Lebesgue integral is indicated.

Step 1: the cost data. By conclusion 1, all entries of $t\mapsto\mathcal{E}_t$, $t\mapsto\mathcal{B}_t$, $t\mapsto\tilde{\mathcal{E}}_t$, $t\mapsto\Theta^\star_t$, $t\mapsto\tilde{\Theta}^\star_t$, $t\mapsto Q_t$, $t\mapsto V_t$, $t\mapsto R_t$, and $t\mapsto R_t^{-1}$ are continuous on $[0,T]$, every $R_t$ is symmetric positive definite, and, with the fluctuation Hessian coefficients $H_{ij}(t)$ and $F_{\gamma\delta}$ of the data,

Exchanging $\gamma$ and $\delta$ leaves these sums unchanged, so $Q^{\gamma\delta}_t=Q^{\delta\gamma}_t$ and $\hat{F}^{\gamma\delta}=\hat{F}^{\delta\gamma}$: every $Q_t$ and $\hat{F}$ is symmetric. Each entry of $V^\circ=\tfrac12V$ is the product of the constant function $\tfrac12$ with a continuous function, hence continuous by the sum and product rules for continuous real-valued functions. Therefore $Q^\circ$ and $R^\circ$ assign symmetric matrices with continuous entries, $V^\circ$ assigns real matrices with $l$ rows and $m$ columns with continuous entries, and $F^\circ$ is symmetric: $(Q^\circ,V^\circ,R^\circ,F^\circ)$ are cost data for the model and the control dimension $k=m$, with every $R^\circ(t)=R_t$ positive definite; and $B$ assigns real matrices with $l$ rows and $m$ columns with continuous entries, as the controlled-system definition requires.

Step 2: model covariances and initial covariance. By the model definition, $\Theta(t)=\varepsilon^\circ(t)\varepsilon^\circ(t)^{\top}$ and $\tilde{\Theta}(t)=\tilde{\varepsilon}^\circ(t)\tilde{\varepsilon}^\circ(t)^{\top}$ --- the model's noise matrices, written $\varepsilon$ and $\tilde{\varepsilon}$ in the model definition, being the present $\varepsilon^\circ$ and $\tilde{\varepsilon}^\circ$ --- so hypothesis (ii) of the statement gives $\Theta(t)=\Theta^\star_t$ and $\tilde{\Theta}(t)=\tilde{\Theta}^\star_t$ for every $t\in[0,T]$. By claim 1 of the Kalman--Bucy filter theorem, the initial covariance matrix of the model is $P^{\mathrm{KB}}_0=\big(\operatorname{Cov}(\xi^{\gamma},\xi^{\delta})\big)_{1\le\gamma,\delta\le l}$ (written $P_0$ in that theorem) with the covariance; by hypothesis (iii) this matrix is $\Pi_0$.

Step 3: $Z$ is a symmetric continuous solution of the backward Riccati equation. By (H2), every $Z_t$ is symmetric, $Z_T=\hat{F}$, and there are continuous functions $\dot{z}^{\gamma\delta}:[0,T]\to\mathbb{R}$ with

in the integral form of the weighted second-moment evolution lemma, with $W_s=Z_s\mathcal{B}_s+\tfrac12V_s$. By claim 4 of the componentwise calculus toolkit, each map $t\mapsto\int_0^t\dot{z}^{\gamma\delta}(s)\,ds$ is continuous on $[0,T]$, so every entry of $Z$ is continuous; and, for $0\le t\le T$,

Subtracting the integral forms at $T$ and at $t$ and using $Z_T=\hat{F}$ therefore gives

Since $V^\circ(s)=\tfrac12V_s$, we have $Z_s\,B(s)+V^\circ(s)=Z_s\mathcal{B}_s+\tfrac12V_s=W_s$ for every $s\in[0,T]$; and $A^\circ(s)=\mathcal{E}_s$, $Q^\circ(s)=Q_s$, $R^\circ(s)=R_s$, $F^\circ=\hat{F}$ (hypothesis (i) and the statement). So, entrywise, the display above is exactly the backward Riccati equation

of the completion-of-squares theorem for the present data. Thus $Z$ is a symmetric continuous solution of that equation, with $Z_t\,B(t)+V^\circ(t)=W_t$: the first part of conclusion 1 of the corollary.

Step 4: the covariance assignment of the Kalman--Bucy filter is $\Pi$. By claim 1 of the Kalman--Bucy filter theorem, formed for the present model, the assignment $D(t)=\tilde{E}^\circ(t)^{\top}\tilde{\Theta}(t)^{-1}\tilde{E}^\circ(t)$ has continuous entries, every $D(t)$ is symmetric positive semidefinite, and there is exactly one assignment $\Pi^{\mathrm{KB}}$ of real matrices with $l$ rows and $l$ columns to the points of $[0,T]$, with continuous entries, such that

this $\Pi^{\mathrm{KB}}$ is the covariance assignment of that claim. By Step 2 and hypothesis (i), $D(t)=\tilde{\mathcal{E}}_t^{\top}(\tilde{\Theta}^\star_t)^{-1}\tilde{\mathcal{E}}_t=\tilde{D}_t$, $A^\circ(r)=\mathcal{E}_r$, $\Theta(r)=\Theta^\star_r$, and $P^{\mathrm{KB}}_0=\Pi_0$; so the displayed equation is the same integral equation, with the same coefficient assignments and the same initial matrix, that the filter covariance $\Pi$ of conclusion 2 satisfies. These data meet the hypotheses of the global existence and uniqueness theorem for the Kalman covariance Riccati equation on $[0,T]$: the entries of $t\mapsto\mathcal{E}_t$, $t\mapsto\Theta^\star_t$, and $t\mapsto\tilde{D}_t$ are continuous and every $\Theta^\star_t$ and every $\tilde{D}_t$ is positive semidefinite, by conclusions 1 and 2, and $\Pi_0$ is positive semidefinite by (H4). Both $\Pi^{\mathrm{KB}}$ and $\Pi$ are assignments with continuous entries satisfying that equation, so the uniqueness assertion of that theorem gives $\Pi^{\mathrm{KB}}(t)=\Pi_t$ for every $t\in[0,T]$: the remaining part of conclusion 1.

Step 5: the optimal value in trace form. Steps 1 and 3 verify all hypotheses of the separation theorem for the present model, the control dimension $m$, the control matrix assignment $B$, the cost data $(Q^\circ,V^\circ,R^\circ,F^\circ)$, and the solution $Z$ of the backward Riccati equation. Its optimal value is

with the trace, the expectation, and $\mathbb{E}[\xi]=(\mathbb{E}[\xi^{1}],\dots,\mathbb{E}[\xi^{l}])$, a Riemann integral of a continuous integrand as recorded there. By hypothesis (iii) every component of $\mathbb{E}[\xi]$ is $0$; hence every component of the matrix-vector product $Z_0\,\mathbb{E}[\xi]$ --- a sum of products each having a factor $0$ --- is $0$, and the dot product $\mathbb{E}[\xi]\cdot(Z_0\,\mathbb{E}[\xi])$ is $0$. Substituting $P^{\mathrm{KB}}_0=\Pi_0$ and $\Theta(t)=\Theta^\star_t$ (Step 2), $Z_t\,B(t)+V^\circ(t)=W_t$ (Step 3), $R^\circ(t)=R_t$, and $\Pi^{\mathrm{KB}}(t)=\Pi_t$ (Step 4):

Step 6: traces as entrywise dot products, and the Lebesgue form. For real matrices $M$ and $N$ with $l$ rows and $l$ columns, claim 4 of the basic properties of the trace gives

Since $Z_0$ and every $Z_t$ are symmetric by (H2), $\operatorname{tr}(Z_0\Pi_0)=\operatorname{tr}(Z_0^{\top}\Pi_0)=Z_0\cdot\Pi_0$ and $\operatorname{tr}(Z_t\,\Theta^\star_t)=\operatorname{tr}(Z_t^{\top}\Theta^\star_t)=Z_t\cdot\Theta^\star_t$. Every $R_t^{-1}$ is symmetric by the invertibility lemma for symmetric positive definite matrices ($R_t$ being symmetric positive definite by conclusion 1), so the transpose identity $(UV)^{\top}=V^{\top}U^{\top}$ of claim 3 of the componentwise calculus toolkit, applied twice, gives

the matrix $W_t\,R_t^{-1}W_t^{\top}$ is symmetric, whence $\operatorname{tr}\big(W_t\,R_t^{-1}W_t^{\top}\,\Pi_t\big)=\operatorname{tr}\big((W_t\,R_t^{-1}W_t^{\top})^{\top}\,\Pi_t\big)=\big(W_t\,R_t^{-1}W_t^{\top}\big)\cdot\Pi_t$. Hence

and the integrand here, agreeing at every $t\in[0,T]$ with the continuous integrand of the display of Step 5, is continuous on $[0,T]$. Its Riemann integral over $[0,T]$ agrees with its Lebesgue integral over the compact interval $[0,T]$, so

whose right-hand side is the right-hand side of the limit identity of the cost-limit proposition.

Step 7: the minimum over extended admissible controls. The hypotheses of the extended separation theorem are exactly those verified in Steps 1 and 3. By its claim 2, $J[\alpha]\ge V^{*}$ for every extended admissible control $\alpha$ with values in $\mathbb{R}^{m}$; by its claim 3, the closed-loop feedback control $\alpha^{*}$ of the closed-loop feedback lemma is admissible, hence extended admissible, and $J[\alpha^{*}]=V^{*}$. Therefore $V^{*}$ is the minimum of $J$ over all extended admissible controls with values in $\mathbb{R}^{m}$, attained by $\alpha^{*}$. Together with Steps 5 and 6 this proves conclusion 2 of the corollary up to its final sentence.

Step 8: the limit identity. Assume finally that for each natural number $N\ge1$ a driving system and a projected solution along the approximate Kalman policy $h^N$ are fixed as in the cost-limit proposition, and that the initial-condition hypotheses (I1)--(I2) there hold. All hypotheses of the cost-limit proposition are then in force --- its setting and (H1)--(H4) are those adopted here --- so its limit identity holds, and by Steps 5--7 its right-hand side equals $V^{*}=\min J$. Hence

which is the final sentence of conclusion 2.

Step 9: realizability of the coefficients (conclusion 3). Set $m^\circ=l\cdot l+\tilde{l}$ and index the $m^\circ$ columns by the $l\cdot l$ ordered pairs $(\sigma,\gamma)\in\{1,\dots,l\}^{2}$ followed by the $\tilde{l}$ channel indices $\upsilon'\in\{1,\dots,\tilde{l}\}$. Write $e_\gamma$ ($\gamma\in\{1,\dots,l\}$) for the $\gamma$-th standard basis vector of Euclidean space $\mathbb{R}^l$, identified with a one-column matrix as in the jump representation lemma; and for a real $x\ge0$ write $\sqrt{x}$ for the unique nonnegative real number with $(\sqrt{x})^{2}=x$, which exists by the existence and uniqueness of the nonnegative square root. Since $S_t$ lies in the probability simplex $\Delta^l$, every $S^\sigma_t\ge0$; every $\beta(\sigma,\gamma,S_t,A_t)\ge0$ by clause 1 of the definition of a transition-rate family; and every $\tilde{b}^{\upsilon'}(S_t)\ge\tilde{r}>0$ by (H3). Define, for $t\in[0,T]$: the matrix $\varepsilon^\circ(t)$, with $l$ rows and $m^\circ$ columns, whose column with index an ordered pair $(\sigma,\gamma)$ with $\sigma\neq\gamma$ is $\sqrt{S^\sigma_t\,\beta(\sigma,\gamma,S_t,A_t)}\,(e_\gamma-e_\sigma)$, and whose remaining columns --- those with index a pair $(\sigma,\sigma)$ and the last $\tilde{l}$ --- are zero; and the matrix $\tilde{\varepsilon}^\circ(t)$, with $\tilde{l}$ rows and $m^\circ$ columns, whose first $l\cdot l$ columns are zero and whose entry in row $\upsilon$ and the column with channel index $\upsilon'$ is $\mathbf{1}_{\{\upsilon=\upsilon'\}}\,\sqrt{\tilde{b}^{\upsilon'}(S_t)}$, with the indicator notation of the LQG data definition.

By the definitions of the matrix product and the transpose, for matrices $M$ and $N$ with $m^\circ$ columns each, the entry of $MN^{\top}$ in row $p$ and column $q$ is $\sum_{c=1}^{m^\circ}M^{pc}N^{qc}$: a sum over the columns, the column $c$ contributing the product of its $p$-th entry in $M$ and its $q$-th entry in $N$. Every nonzero column of $\varepsilon^\circ(t)$ has index among the first $l\cdot l$ and every nonzero column of $\tilde{\varepsilon}^\circ(t)$ has index among the last $\tilde{l}$, so in $\varepsilon^\circ(t)\,\tilde{\varepsilon}^\circ(t)^{\top}$ every term of every entry has a factor $0$: $\varepsilon^\circ(t)\,\tilde{\varepsilon}^\circ(t)^{\top}=0$, the zero matrix with $l$ rows and $\tilde{l}$ columns. In $\varepsilon^\circ(t)\,\varepsilon^\circ(t)^{\top}$, the column with index $(\sigma,\gamma)$, $\sigma\neq\gamma$, contributes to the entry in row $p$ and column $q$ the term $S^\sigma_t\,\beta(\sigma,\gamma,S_t,A_t)\,\big((e_\gamma-e_\sigma)(e_\gamma-e_\sigma)^{\top}\big)^{pq}$ --- the square of the square root being its argument --- and the remaining columns contribute $0$; hence, summing over the ordered pairs and applying clause 1 (the jump representation) of the jump representation and positive semidefiniteness of the aggregate fluctuation covariance at the point $(S_t,A_t)\in\Delta^l\times\mathbb{R}^m$,

the last equality by clause 6 of the LQG data definition. In $\tilde{\varepsilon}^\circ(t)\,\tilde{\varepsilon}^\circ(t)^{\top}$, the entry in row $\upsilon$ and column $\upsilon''$ is $\sum_{\upsilon'=1}^{\tilde{l}}\mathbf{1}_{\{\upsilon=\upsilon'\}}\mathbf{1}_{\{\upsilon''=\upsilon'\}}\,\tilde{b}^{\upsilon'}(S_t)=\mathbf{1}_{\{\upsilon=\upsilon''\}}\,\tilde{b}^{\upsilon}(S_t)$, which is the corresponding entry of $\tilde{\Theta}^\star_t$ by clause 7 of the LQG data definition: $\tilde{\varepsilon}^\circ(t)\,\tilde{\varepsilon}^\circ(t)^{\top}=\tilde{\Theta}^\star_t$.

For the continuity of the entries: every entry of $t\mapsto\varepsilon^\circ(t)$ and of $t\mapsto\tilde{\varepsilon}^\circ(t)$ is either constantly $0$ or of the form $\pm\sqrt{g}$ for the nonnegative function $g(t)=S^\sigma_t\,\beta(\sigma,\gamma,S_t,A_t)$, respectively $g(t)=\tilde{b}^{\upsilon'}(S_t)$. The map $t\mapsto\tilde{b}^{\upsilon'}(S_t)$ is continuous by conclusion 1. The components of $t\mapsto(S_t,A_t)$ are continuous by clause 1 of the definition of a mean-field trajectory pair (part of the stationary triple setting); on $\Delta^l\times\mathbb{R}^m$ the rate $\beta(\sigma,\gamma,\cdot,\cdot)$ agrees with the restriction of $\bar{\beta}(\sigma,\gamma,\cdot,\cdot)$ by clause 1 of the definition of the transition-rate extension, and $\bar{\beta}(\sigma,\gamma,\cdot,\cdot)$ is a $C^1$ map on $U\times\mathbb{R}^m$ by clause 2 of that definition, hence continuous; so $t\mapsto\beta(\sigma,\gamma,S_t,A_t)$ is continuous on $[0,T]$ by continuity of compositions along the continuous map $t\mapsto(S_t,A_t)$, applied pointwise on $[0,T]$ exactly as in the definition of the fluctuation linear-quadratic cost, and $t\mapsto S^\sigma_t\,\beta(\sigma,\gamma,S_t,A_t)$ is continuous by the sum and product rules for continuous real-valued functions. Finally, the nonnegative square root preserves continuity. For reals $0\le u\le v$ one has $\sqrt{u}\le\sqrt{v}$, and for $0\le u<v$ one has $\sqrt{u}<\sqrt{v}$: otherwise $\sqrt{u}\ge\sqrt{v}\ge0$, respectively $\sqrt{u}>\sqrt{v}\ge0$, would give $u=(\sqrt{u})^{2}\ge(\sqrt{v})^{2}=v$, respectively $u>v$. For reals $x,y\ge0$, $|\sqrt{x}-\sqrt{y}|\le\sqrt{x}+\sqrt{y}$, both square roots being nonnegative, so

the middle equality by expanding the product of the difference and the sum of $\sqrt{x}$ and $\sqrt{y}$; since $|\sqrt{x}-\sqrt{y}|$ is the nonnegative square root of its own square, the monotonicity above yields $|\sqrt{x}-\sqrt{y}|\le\sqrt{|x-y|}$. Hence if $g:[0,T]\to\mathbb{R}$ is continuous with $g\ge0$ and $t_0\in[0,T]$, then for every real $\eta>0$ there is a $\delta>0$ such that $|g(t)-g(t_0)|<\eta^{2}$ for all $t\in[0,T]$ with $|t-t_0|<\delta$, and then $|\sqrt{g(t)}-\sqrt{g(t_0)}|\le\sqrt{|g(t)-g(t_0)|}<\eta$ by the strict monotonicity above: the map $t\mapsto\sqrt{g(t)}$ is continuous on $[0,T]$. Therefore all entries of $\varepsilon^\circ$ and $\tilde{\varepsilon}^\circ$ are continuous on $[0,T]$; and since every $\tilde{\Theta}^\star_t$ is symmetric positive definite by conclusion 1, the displayed identities show that these assignments satisfy hypothesis (ii) of the statement and conditions (i) and (ii) of the model definition. This proves conclusion 3. $\blacksquare$