The Approximate Kalman Filter and Policy for the Controlled N-Agent Dynamics

Adopt the setting and notation of the definition of the fluctuation LQG data of a stationary mean-field triple: natural numbers $l\ge2$, $m\ge1$, $\tilde{l}\ge1$, the control set $\mathcal{A}$ (a nonempty subset of $\mathbb{R}^m$), the transition-rate family $\beta$ with control set $\mathcal{A}$ and rate bound $B$ and its extension $(U,V,\bar{\beta})$, the cost extension — whose open set, written $W$ in the LQG-data setting, we here write $U_c$, so the cost extension is $(U_c,\bar{L},\bar{G})$, freeing the letter $W$; the time-indexed matrices $V_t$ and $W_t$ below are unrelated to the control-side open set $V$ of $(U,V,\bar{\beta})$ — the horizon $T>0$, the stationary mean-field triple $(S,A,P)$, the observation-rate family $\tilde{\beta}$ with $\tilde{l}$ channels and rate bound $\tilde{B}$ and its extension $(\tilde{U},\bar{\tilde{\beta}})$, and the resulting matrices $\mathcal{E}_t$, $\mathcal{B}_t$, $\tilde{\mathcal{E}}_t$, $H^{SS}_t$, $H^{SA}_t$, $H^{AS}_t$, $H^{AA}_t$, $F^\star$, $\Theta^\star_t$, $\tilde{\Theta}^\star_t$ ($t\in[0,T]$). Let $\tilde{b}$ be the aggregate observation drift of $\tilde{\beta}$, write $\tilde{b}(\Sigma)$ for the vector with components $\tilde{b}^1(\Sigma),\dots,\tilde{b}^{\tilde{l}}(\Sigma)$, and write $\Delta^l$ for the probability simplex. Vectors of $\mathbb{R}^l$, $\mathbb{R}^m$, and $\mathbb{R}^{\tilde{l}}$ are identified with one-column matrices, products of matrices are matrix products, sums of matrices are entrywise, $(\cdot)^{\top}$ is the transpose, $I$ is the identity matrix of the indicated size, inverses of square matrices are matrix inverses, $|\cdot|$ is the Euclidean norm (Euclidean distance to the origin), and $e_\upsilon$ ($\upsilon\in\{1,\dots,\tilde{l}\}$) is the $\upsilon$-th standard basis vector of Euclidean space $\mathbb{R}^{\tilde{l}}$. Throughout, a real-valued function on a subinterval $I$ of the real numbers $\mathbb{R}$ is called continuous on $I$ when it is continuous relative to $I$, both $I$ and the codomain $\mathbb{R}$ carrying the metric of the real line. All Riemann integrals of matrix- or vector-valued maps below are entrywise Riemann integrals over compact intervals of continuous integrands, which exist and agree with the corresponding Lebesgue integrals by claim 3 of the interval toolkit, and are $0$ over degenerate intervals. Define, for $t\in[0,T]$, the symmetrized coefficient matrices

$$Q_t=\tfrac{1}{4}\big(H^{SS}_t+(H^{SS}_t)^{\top}\big),\qquad V_t=\tfrac{1}{2}\big(H^{SA}_t+(H^{AS}_t)^{\top}\big),\qquad R_t=\tfrac{1}{4}\big(H^{AA}_t+(H^{AA}_t)^{\top}\big),\qquad \hat{F}=\tfrac{1}{4}\big(F^\star+(F^\star)^{\top}\big);$$

conclusion 1 below records their entries in terms of the fluctuation Hessian coefficients, matching the defining formulas of the coefficient matrices of the completion-of-squares theorem. Assume:

(H1) there is a real $r>0$ such that $\sum_{i=1}^{m}\sum_{j=1}^{m}R^{ij}_t\,a^i\,a^j\ge r\,|a|^2$ for every $t\in[0,T]$ and every $a\in\mathbb{R}^m$;

(H2) there is a family $Z=(Z_t)_{t\in[0,T]}$ of symmetric real matrices with $l$ rows and $l$ columns, continuously differentiable in integral form as in the weighted second-moment evolution lemma, with terminal value $Z_T=\hat{F}$ and densities

$$\dot{z}^{\gamma\delta}(t)=-\big(\mathcal{E}_t^{\top}Z_t+Z_t\,\mathcal{E}_t-W_t\,R_t^{-1}W_t^{\top}+Q_t\big)^{\gamma\delta}\qquad\text{with}\qquad W_t=Z_t\,\mathcal{B}_t+\tfrac{1}{2}V_t$$

— the backward Riccati equation of hypothesis (H2) of the completion-of-squares theorem; here $R_t^{-1}$ exists by conclusion 1 below, whose proof uses only (H1);

(H3) there is a real $\tilde{r}>0$ such that $\tilde{b}^\upsilon(S_t)\ge\tilde{r}$ for every $\upsilon\in\{1,\dots,\tilde{l}\}$ and every $t\in[0,T]$;

(H4) $\Pi_0$ is a symmetric positive semidefinite real matrix with $l$ rows and $l$ columns;

(C) the control set $\mathcal{A}$ is closed: its complement $\mathbb{R}^m\setminus\mathcal{A}$ is an open subset of $\mathbb{R}^m$.

Then:

1. (Coefficients.) All entries of the maps $t\mapsto\mathcal{E}_t$, $t\mapsto\mathcal{B}_t$, $t\mapsto\tilde{\mathcal{E}}_t$, $t\mapsto\Theta^\star_t$, $t\mapsto\tilde{\Theta}^\star_t$, $t\mapsto Q_t$, $t\mapsto V_t$, $t\mapsto R_t$, and $t\mapsto\tilde{b}(S_t)$ are continuous on $[0,T]$. With the fluctuation Hessian coefficients $H_{ij}(t)$ and $F_{\gamma\delta}$ of these data, for all indices:

$$Q^{\gamma\delta}_t=\tfrac{1}{4}\big(H_{\gamma\delta}(t)+H_{\delta\gamma}(t)\big),\qquad V^{\gamma j}_t=\tfrac{1}{2}\big(H_{\gamma,l+j}(t)+H_{l+j,\gamma}(t)\big),\qquad R^{ij}_t=\tfrac{1}{4}\big(H_{l+i,l+j}(t)+H_{l+j,l+i}(t)\big),\qquad \hat{F}^{\gamma\delta}=\tfrac{1}{4}\big(F_{\gamma\delta}+F_{\delta\gamma}\big)$$

— the defining formulas of the coefficient matrices of the completion-of-squares theorem. Each $R_t$ is symmetric positive definite, hence invertible; each $\tilde{\Theta}^\star_t$ is symmetric positive definite, and its inverse is the diagonal matrix whose diagonal entries are $1/\tilde{b}^1(S_t),\dots,1/\tilde{b}^{\tilde{l}}(S_t)$; and all entries of $t\mapsto R_t^{-1}$, of $t\mapsto(\tilde{\Theta}^\star_t)^{-1}$, and of the feedback gain $t\mapsto\mathcal{G}_t=R_t^{-1}W_t^{\top}$ ($m$ rows, $l$ columns; this letter is unrelated to the observation filtration written $(\mathcal{G}_t)$ in the solution definition used in conclusion 4) are continuous on $[0,T]$.

2. (Filter covariance and gains.) Each $\tilde{D}_t=\tilde{\mathcal{E}}_t^{\top}(\tilde{\Theta}^\star_t)^{-1}\tilde{\mathcal{E}}_t$ is symmetric positive semidefinite with entries continuous in $t$, and each $\Theta^\star_t$ is positive semidefinite by the jump representation of the aggregate fluctuation covariance. Consequently, by the global existence and uniqueness theorem for the Kalman covariance Riccati equation, there is exactly one assignment $\Pi$ of a real matrix $\Pi_t$ with $l$ rows and $l$ columns to each $t\in[0,T]$, with continuous entries, such that

$$\Pi_t=\Pi_0+\int_0^t\big(\mathcal{E}_r\,\Pi_r+\Pi_r\,\mathcal{E}_r^{\top}-\Pi_r\,\tilde{D}_r\,\Pi_r+\Theta^\star_r\big)\,dr\qquad(0\le t\le T),$$

called the filter covariance of these data, and every $\Pi_t$ is symmetric positive semidefinite. All entries of the Kalman gain $t\mapsto\tilde{\mathcal{K}}_t=\Pi_t\,\tilde{\mathcal{E}}_t^{\top}(\tilde{\Theta}^\star_t)^{-1}$ ($l$ rows, $\tilde{l}$ columns) and of the closed-loop matrix $t\mapsto M_t=\mathcal{E}_t-\mathcal{B}_t\,\mathcal{G}_t-\tilde{\mathcal{K}}_t\,\tilde{\mathcal{E}}_t$ ($l$ rows, $l$ columns) are continuous on $[0,T]$.

3. (The approximate Kalman policy.) Let $\Phi$ and $\Psi=\Phi^{-1}$ be the fundamental solution of $M$ on $[0,T]$ and its inverse from the fundamental-solution theorem. Fix a natural number $N\ge1$. For $k\ge1$ write $\mathcal{R}_k$ for the record space with horizon $T$ (denoted there with the letter $R$) of the definition of an observation-driven control policy, and define, for $t\in[0,T]$, $\tau=(\tau_1,\dots,\tau_k)\in\mathcal{R}_k$, and $v=(v_1,\dots,v_k)\in\{1,\dots,\tilde{l}\}^k$,

$$f^N_k(t,\tau,v)=\Phi(t)\Big(-N^{1/2}\int_0^t\Psi(r)\,\tilde{\mathcal{K}}_r\,\tilde{b}(S_r)\,dr+N^{-1/2}\sum_{j=1}^{k}\mathbf{1}_{\{\tau_j\le t\}}\,\Psi(\tau_j)\,\tilde{\mathcal{K}}_{\tau_j}\,e_{v_j}\Big)\in\mathbb{R}^l,$$

where $\mathbf{1}_{\{\tau_j\le t\}}$ equals $1$ if $\tau_j\le t$ and $0$ otherwise; define $f^N_0(t)$ by the same formula with the sum omitted, and define (the Kalman clamp: the candidate control is kept only when it lies in the control set)

$$h^N_k(t,\tau,v)=\begin{cases}A_t-N^{-1/2}\,\mathcal{G}_t\,f^N_k(t,\tau,v)&\text{if }A_t-N^{-1/2}\,\mathcal{G}_t\,f^N_k(t,\tau,v)\in\mathcal{A},\\ A_t&\text{otherwise,}\end{cases}\qquad(k\ge1),$$

and $h^N_0(t)\in\mathbb{R}^m$ by the same two-case formula with $f^N_0(t)$ in place of $f^N_k(t,\tau,v)$. Then $h^N=(h^N_k)_{k\ge0}$ is an observation-driven control policy with horizon $T$, control dimension $m$, and $\tilde{l}$ channels, and it is $\mathcal{A}$-valued; it is called the approximate Kalman policy of these data at level $N$. Also $f^N=(f^N_k)_{k\ge0}$ is an observation-driven control policy with horizon $T$, control dimension $l$, and $\tilde{l}$ channels. Moreover the family $\beta^{\sharp}$ of functions on $\Delta^l\times(\mathcal{A}\times\mathbb{R}^l)$ defined by $\beta^{\sharp}(\sigma,\gamma,\Sigma,(a,x))=\beta(\sigma,\gamma,\Sigma,a)$ for $a\in\mathcal{A}$ and $x\in\mathbb{R}^l$ (points of $\mathbb{R}^{m+l}$ being split as pairs, under which $\mathcal{A}\times\mathbb{R}^l$ is a nonempty subset of $\mathbb{R}^{m+l}$) is a transition-rate family on $l$ states with control set $\mathcal{A}\times\mathbb{R}^l$ and rate bound $B$, and the family $h^{\sharp}=(h^{\sharp}_k)_{k\ge0}$ with $h^{\sharp}_k(t,\tau,v)=\big(h^N_k(t,\tau,v),f^N_k(t,\tau,v)\big)\in\mathbb{R}^{m+l}$ and $h^{\sharp}_0(t)=\big(h^N_0(t),f^N_0(t)\big)$ is an observation-driven control policy with horizon $T$, control dimension $m+l$, and $\tilde{l}$ channels, and it is $(\mathcal{A}\times\mathbb{R}^l)$-valued.

4. (Realization along the controlled dynamics.) Let $(\Omega,\mathcal{F},\mathbb{P})$ be an $N$-agent driving system with $l$ states and $\tilde{l}$ observation channels (its probability measure written $\mathbb{P}$, the letter $P$ denoting the stationary co-state), and let state processes $\sigma^i$, observation processes $\Upsilon^\upsilon$, a control process $\alpha^{\sharp}$ with values in $\mathcal{A}\times\mathbb{R}^l$, and a regular event $\Omega_0$ be a solution of the controlled $N$-agent dynamics on $[0,T]$ for $\beta^{\sharp}$, $\tilde{\beta}$, this driving system, and the policy $h^{\sharp}$; such solutions exist by the existence and uniqueness theorem for the controlled $N$-agent dynamics. Write $\alpha_t$ for the vector of the first $m$ components of $\alpha^{\sharp}_t$, called the approximate Kalman control, and $\hat{\mathfrak{s}}^N_t$ for the vector of the last $l$ components of $\alpha^{\sharp}_t$, called the approximate Kalman filter; and at each $\omega\in\Omega_0$ let $\tilde{c}_t$ be the observation total and $\tau_1<\dots<\tau_{\tilde{c}_T}$ and $\upsilon_1,\dots,\upsilon_{\tilde{c}_T}$ the jump times of the observation total and their channels, as in condition 5 of the definition of a solution (where $\tilde{c}_t$ is written $K_t$). Then:

(a) (Projection and policy identity.) The state processes $\sigma^i$, the observation processes $\Upsilon^\upsilon$, the control process $\alpha$, and the regular event $\Omega_0$ are a solution of the controlled $N$-agent dynamics on $[0,T]$ for $\beta$, $\tilde{\beta}$, the same driving system, and the policy $h^N$; and conversely, if a collection of state processes, observation processes, an $\mathcal{A}$-valued control process, and a regular event is a solution for $\beta$, $\tilde{\beta}$, this driving system, and $h^N$, then it is indistinguishable from the projected solution above in the sense of the uniqueness theorem. At every $\omega\in\Omega_0$ and every $t\in[0,T]$:

$$\hat{\mathfrak{s}}^N_t=f^N_{\tilde{c}_t}\big(t,(\tau_1,\dots,\tau_{\tilde{c}_t}),(\upsilon_1,\dots,\upsilon_{\tilde{c}_t})\big),\qquad \hat{\mathfrak{s}}^N_0=0,$$

and, writing $\chi_t(\omega)=1$ if $A_t-N^{-1/2}\,\mathcal{G}_t\,\hat{\mathfrak{s}}^N_t(\omega)\in\mathcal{A}$ and $\chi_t(\omega)=0$ otherwise — the clamp indicator —

$$\alpha_t=\chi_t\,\big(A_t-N^{-1/2}\,\mathcal{G}_t\,\hat{\mathfrak{s}}^N_t\big)+(1-\chi_t)\,A_t,\qquad\text{equivalently}\qquad N^{1/2}\,(\alpha_t-A_t)=-\chi_t\,R_t^{-1}\,W_t^{\top}\,\hat{\mathfrak{s}}^N_t.$$

(b) (Filter equation.) At every $\omega\in\Omega_0$ and every $t\in[0,T]$, componentwise:

$$\hat{\mathfrak{s}}^N_t=\int_{[0,t]}\Big(\mathcal{E}_r\,\hat{\mathfrak{s}}^N_r-\mathcal{B}_r\,\mathcal{G}_r\,\hat{\mathfrak{s}}^N_r-\tilde{\mathcal{K}}_r\,\big(N^{1/2}\,\tilde{b}(S_r)+\tilde{\mathcal{E}}_r\,\hat{\mathfrak{s}}^N_r\big)\Big)\,dr+N^{-1/2}\sum_{j=1}^{\tilde{c}_t}\tilde{\mathcal{K}}_{\tau_j}\,e_{\upsilon_j},$$

where for $t>0$ the integral is the Lebesgue integral over the compact interval $[0,t]$ of a bounded measurable integrand whose components are continuous at every $r$ other than the finitely many jump times, the integral is $0$ for $t=0$ (so that the identity then reads $\hat{\mathfrak{s}}^N_0=0$), and the sum is $0$ when $\tilde{c}_t=0$. The drift term $-\mathcal{B}_r\,\mathcal{G}_r\,\hat{\mathfrak{s}}^N_r$ coincides with $N^{1/2}\,\mathcal{B}_r\,(\alpha_r-A_r)$ exactly where $\chi_r=1$: the filter recursion is autonomous in the observations and is unaffected by the clamp.

(c) (Path regularity, measurability, and bounds.) At every $\omega\in\Omega_0$, every component path $t\mapsto\hat{\mathfrak{s}}^{N,\gamma}_t$ ($\gamma\in\{1,\dots,l\}$) is right-continuous on $[0,T]$ and continuous at every $t$ that is not one of $\tau_1,\dots,\tau_{\tilde{c}_T}$. Each map $(t,\omega)\mapsto\mathbf{1}_{\Omega_0}(\omega)\,\hat{\mathfrak{s}}^{N,\gamma}_t(\omega)$ is measurable with respect to the product $\sigma$-algebra of the trace Borel $\sigma$-algebra on $[0,T]$ and $\mathcal{F}$, where $\mathbf{1}_{\Omega_0}$ equals $1$ on $\Omega_0$ and $0$ off $\Omega_0$. Moreover there is a real $C^\circ\ge0$, determined by $l$, $\tilde{l}$, $m$, $T$, $\tilde{B}$, and the entry bounds of $\Phi$, $\Psi$, $\tilde{\mathcal{K}}$, and $\mathcal{G}$ on $[0,T]$ — in particular the same for every $N$, every driving system, and every solution — such that at every $\omega\in\Omega_0$ and every $t\in[0,T]$:

$$|\hat{\mathfrak{s}}^N_t|\le C^\circ\big(N^{1/2}+N^{-1/2}\,\tilde{c}_T\big)\qquad\text{and}\qquad N^{1/2}\,|\alpha_t-A_t|\le C^\circ\big(N^{1/2}+N^{-1/2}\,\tilde{c}_T\big).$$