Proof of Cost Limit Along the Approximate Kalman Policy

Throughout, fix the fluctuation LQG data and, for each $N$, the driving system and projected solution of the statement; superscripts $N$ are suppressed when no confusion can arise. Write $\mathbb{E}$ for the expectation of the $N$-th probability space, $z_t=(\mathfrak{s}_t,\mathfrak{a}_t)\in\mathbb{R}^{l+m}$, $\rho_t=N^{-1/2}|z_t|$, and $\kappa=1+\sup_N\mathbb{E}[|\mathfrak{s}^N_0|^4]$, finite by (I2). Fix, by conclusion 1 of the policy lemma and the extreme value theorem (each continuous entry attaining a maximum and a minimum on $[0,T]$, hence being bounded in absolute value), entry bounds $C_Z$ for $Z_t$, $c_W$ for $W_tR_t^{-1}W_t^{\top}$, $c_R$ for $R_t$, and $C_{\mathcal{G}}$ for the feedback gain $\mathcal{G}_t=R_t^{-1}W_t^{\top}$ on $[0,T]$, and write $g^*=\sqrt{ml}\,C_{\mathcal{G}}$ for the resulting operator bound of $\mathcal{G}_t$ (the row-wise Cauchy-Schwarz estimate $|Hx|\le\sqrt{pq}\,c|x|$ for a matrix $H$ with $p$ rows, $q$ columns and entries bounded by $c$, as in the proof of the completion-of-squares theorem); the entries of $W_tR_t^{-1}W_t^{\top}$ are continuous, as finite sums of products of the continuous entries of $Z$, $\mathcal{B}$, $V$, $R^{-1}$ (conclusion 1 of the policy lemma and (H2)). By conclusion 1 of the policy lemma, its matrices $Q_t$, $V_t$, $R_t$, $W_t$, and $\hat{F}$ are given by the same defining formulas as those of the completion-of-squares theorem, and the family $Z$ of (H2) satisfies the Riccati equation of hypothesis (H2) of that theorem; so both that theorem and the cost expansion theorem are available for each fixed $N$, their common hypothesis $\mathcal{A}_2=\int_{[0,T]}\mathbb{E}[|\mathfrak{a}_t|^2]dt<\infty$ holding by part (a) of the filter error covariance lemma. By part (d) of that lemma there is a real $C_1$, independent of $N$, with

for every $N$ (the second bound obtained from the first, together with $\mathbb{E}[|\cdot|^2]\le1+\mathbb{E}[|\cdot|^4]$ applied to the four summands of the first display and $\kappa\ge1$, as in part (d) of that lemma); in particular $\mathbb{E}[|z_t|^2]\le2(4+C_1)\kappa$ and $\mathbb{E}[|z_t|^4]\le8(\mathbb{E}[|\mathfrak{s}_t|^4]+\mathbb{E}[|\mathfrak{a}_t|^4])\le16C_1\kappa$ for all $t$ and $N$. Finally, the limit integrand $t\mapsto Z_t\cdot\Theta^\star_t+(W_tR_t^{-1}W_t^{\top})\cdot\Pi_t$ is continuous, being a finite sum of products of continuous functions (conclusions 1 and 2 of the policy lemma and (H2)), which proves the first assertion.

Step 1: expansion and completion of squares. Fix $N$. By parts (b) and (c) of the cost expansion theorem, $J^N[h^N]$ is finite, $LQG[(\mathfrak{s}),(\mathfrak{a})]$ is a well-defined real number, and

with $R_N$ obeying the modulus bound of its part (c). By part (c) of the completion-of-squares theorem, with $u_t=\mathfrak{a}_t+R_t^{-1}W_t^{\top}\mathfrak{s}_t$ and $e_s$ its linearization residual,

all terms finite. In particular each $N(J^N[h^N]-J^{MF})+\sum_\gamma P^\gamma_0\zeta^\gamma_N$ is a well-defined real number. We treat the four pieces in Steps 2--5.

Step 2: initial term. $\mathbb{E}[\mathfrak{s}_0\cdot Z_0\mathfrak{s}_0]=\sum_{\gamma,\delta}Z^{\gamma\delta}_0\,\mathbb{E}[\mathfrak{s}^{N,\gamma}_0\mathfrak{s}^{N,\delta}_0]\to\sum_{\gamma,\delta}Z^{\gamma\delta}_0\Pi^{\gamma\delta}_0=Z_0\cdot\Pi_0$ as $N\to\infty$, by (I1) and linearity (a finite sum of convergent sequences).

Step 3: control term. By conclusion 4(a) of the policy lemma, $\mathfrak{a}_t=N^{1/2}(\alpha_t-A_t)=-\chi_t\,R_t^{-1}W_t^{\top}\hat{\mathfrak{s}}^N_t$ on the regular event, so that almost surely

the clamp contribution to the control, which vanishes wherever the clamp is inactive.

The clamp contribution is negligible. If $g^*=0$ then $\mathcal{G}_s\hat{\mathfrak{s}}^N_s=0$, so $q_s=0$ and the two correction terms below vanish; assume therefore $g^*>0$. At every point of the regular event at which $\chi_s=0$ the point $A_s-N^{-1/2}\mathcal{G}_s\hat{\mathfrak{s}}^N_s$ lies outside $\mathcal{A}$, so by (H5) its Euclidean distance to $A_s$ exceeds $\varrho$, that is $N^{-1/2}|\mathcal{G}_s\hat{\mathfrak{s}}^N_s|>\varrho$; with $|\mathcal{G}_s\hat{\mathfrak{s}}^N_s|\le g^*|\hat{\mathfrak{s}}^N_s|$ this gives $1<(g^*)^2\varrho^{-2}N^{-1}|\hat{\mathfrak{s}}^N_s|^2$ there. Multiplying by $|\hat{\mathfrak{s}}^N_s|^2$, and observing that where $\chi_s=1$ the left side below is $0$ while the right side is nonnegative, we obtain on the regular event the pointwise bound $(1-\chi_s)|\hat{\mathfrak{s}}^N_s|^2\le(g^*)^2\varrho^{-2}N^{-1}|\hat{\mathfrak{s}}^N_s|^4$, whence by monotonicity of the integral and (0.1), using $(1-\chi_s)^2=1-\chi_s$,

for every $s$ and $N$. Therefore, for every $s$, expanding the quadratic form and using $u_s=\mathcal{G}_s\varepsilon^N_s+q_s$,

where the leading term is computed entrywise,

where the last equality uses $\mathcal{G}_s=R_s^{-1}W_s^{\top}$, the symmetry of $R_s$ and of $R_s^{-1}$ (invertibility of symmetric positive definite matrices), and $R_sR_s^{-1}=I$: $\mathcal{G}^{\top}R\mathcal{G}=WR^{-1}RR^{-1}W^{\top}=WR^{-1}W^{\top}$, with the reversal rule for the transpose. Write $\eta_N$ for the supremum over $s\in[0,T]$ of $\big|2\mathbb{E}[\mathcal{G}_s\varepsilon^N_s\cdot R_sq_s]+\mathbb{E}[q_s\cdot R_sq_s]\big|$. By the row-wise estimate $|x\cdot R_sy|\le m\,c_R|x||y|$, the Cauchy-Schwarz inequality, the bound $\mathbb{E}[|\varepsilon^N_s|^2]\le1+\mathbb{E}[|\varepsilon^N_s|^4]\le(1+C_1)\kappa$ from (0.1) with $\kappa\ge1$, and (3.1),

Consequently, by linearity and monotonicity of the integral (all integrands bounded and measurable in $s$: part (a) of the filter error covariance lemma for $\Pi^N$, continuity for $\Pi$, and the two correction terms bounded by $\eta_N$ uniformly in $s$),

By part (e) of the filter error covariance lemma, the supremum is at most $C(\max_{\gamma,\delta}|\mathbb{E}[\mathfrak{s}^{N,\gamma}_0\mathfrak{s}^{N,\delta}_0]-\Pi^{\gamma\delta}_0|+N^{-1/2}(1+\mathbb{E}[|\mathfrak{s}^N_0|^4]))\le C(\max_{\gamma,\delta}|\mathbb{E}[\mathfrak{s}^{N,\gamma}_0\mathfrak{s}^{N,\delta}_0]-\Pi^{\gamma\delta}_0|+N^{-1/2}\kappa)$, which tends to $0$ by (I1) and (I2); and $T\eta_N\to0$ as just shown. Hence $\int\mathbb{E}[u_s\cdot R_su_s]ds\to\int_{[0,T]}(W_sR_s^{-1}W_s^{\top})\cdot\Pi_s\,ds$.

Step 4: covariance term. By clause 6 of the fluctuation LQG data, $(\Theta^\star_s)^{\gamma\delta}=\Theta^{\gamma\delta}(S_s,A_s)$. By part (c) of the covariance deviation lemma together with (0.1) and monotonicity of the integral,

with the constant $c_\Theta$ of that lemma and $\mathcal{S}+\mathcal{A}_2\le2T(4+C_1)\kappa$ by (0.1). This tends to $0$, and $\int_{[0,T]}\sum_{\gamma,\delta}Z^{\gamma\delta}_s\Theta^{\gamma\delta}(S_s,A_s)ds=\int_{[0,T]}Z_s\cdot\Theta^\star_s\,ds$ by the definition of the pairing.

Step 5: residual term. By part (b) of the completion-of-squares theorem, $|e_s|\le c_eN^{-1/2}(|\mathfrak{s}_s|^2+|\mathfrak{a}_s|^2)=c_eN^{-1/2}|z_s|^2$ with $c_e=\tfrac32l^{3/2}(l+m)K$, the constant $K$ being the derivative bound of the transition-rate extension and $K_c$ the second-derivative bound of the cost extension, as in the cost expansion theorem. Since $|x\cdot Z_sy|\le C_Z(\sum_\gamma|x^\gamma|)(\sum_\delta|y^\delta|)\le lC_Z|x||y|$, and pointwise $|\mathfrak{s}_s|\,|z_s|^2\le|z_s|^3\le1+|z_s|^4$,

Step 6: the remainder vanishes. Let $\theta>0$. By part (a) of the cost expansion theorem there is $\delta^\circ>0$ with $\omega_L(u)\le\theta$, $\omega_b(u)\le\theta$, and $\omega_G(u)\le\theta$ for all $u\in[0,\delta^\circ]$, and globally $\omega_L\le2K_c$, $\omega_b\le6lK$, $\omega_G\le2K_c$. Let $C_P$ be the co-state bound of that theorem and write $\bar{\omega}=2K_c+C_P\,6lK$. Splitting the time integral of its part (c) on the events $\{\rho_t\le\delta^\circ\}$ and $\{\rho_t>\delta^\circ\}$ (the moduli being nondecreasing) and using monotonicity,

where $\mathbf{1}_E$ denotes the indicator of the event $E$ and we used $d(\Sigma_T,S_T)=N^{-1/2}|\mathfrak{s}_T|$. By the Cauchy-Schwarz inequality and Markov's inequality applied to the nonnegative variable $|z_t|^2$ at level $N(\delta^\circ)^2$,

using $\sqrt{C_1(4+C_1)}\le4+C_1$ (since $C_1\le4+C_1$) and $4\sqrt{2}\le6$; using (0.1), and the same bound holds for the terminal term with $|\mathfrak{s}_T|$ in place of $|z_t|$. Hence, with (0.1) again,

and since $\theta>0$ was arbitrary, $R_N\to0$.

Step 7: conclusion. Combining Steps 1--6, the sequence $N(J^N[h^N]-J^{MF})+\sum_\gamma P^\gamma_0\zeta^\gamma_N$ is, for each $N$, the sum of four terms converging respectively to $Z_0\cdot\Pi_0$, $\int_{[0,T]}(W_sR_s^{-1}W_s^{\top})\cdot\Pi_s\,ds$, $\int_{[0,T]}Z_s\cdot\Theta^\star_s\,ds$, and $0$, plus the remainder $R_N\to0$. The limit identity follows, the two integrals combining by linearity of the integral into the integral of the continuous integrand recorded in the statement. $\blacksquare$