Proof of Mean-Square Tracking of the Mean-Field Trajectory under an Open-Loop Control

Let $\overline{M^\gamma}$ and $\overline{M}$ be as in the martingale bound for the empirical state measure for the given event $\Omega_*$, so that $P(\Omega_*)=1$, $\Omega_*\subseteq\Omega_0$ with $\Omega_0$ the regular event, and $|M_t(\omega)|\le\overline{M}(\omega)$ for every $t\in[0,T]$ and $\omega\in\Omega_*$.

Step 1: $\Psi$ is a random variable. Both $\Sigma_t$ and $S_t$ lie in the probability simplex, so $|\Sigma_t|\le1$ and $|S_t|\le1$, whence $|\Sigma_t-S_t|\le2$ for every $t$ and every $\omega$, by the triangle inequality. For $\omega\in\Omega_*$ each path $t\mapsto\Sigma^\gamma_t(\omega)$ is right-continuous at every $t\in[0,T)$ by the martingale bound for the empirical state measure, hence so is $t\mapsto\Sigma_t(\omega)$, and $t\mapsto S_t$ is continuous; hence $t\mapsto\Sigma_t(\omega)-S_t$ is right-continuous there, and so is $t\mapsto|\Sigma_t(\omega)-S_t|$. Applying the supremum lemma for bounded right-continuous processes to the family of random variables $Z_t=|\Sigma_t-S_t|$ with $K=2$ and the event $\Omega_*$, the map $\Psi$ is a random variable with $0\le\Psi\le2$, and $\Psi(\omega)=\sup_{t\in[0,T]}|\Sigma_t(\omega)-S_t|$ for every $\omega\in\Omega_*$.

Step 2: a pathwise integral inequality. Fix $\omega\in\Omega_*$ and write $\Psi_t=\sup_{s\in[0,t]}|\Sigma_s(\omega)-S_s|$ for $t\in[0,T]$, so that $\Psi_T=\Psi(\omega)$ by Step 1. The function $t\mapsto\Psi_t$ is nondecreasing with values in $[0,2]$, hence bounded and measurable by measurability of monotone functions.

By the open-loop policy lemma, $\alpha_s(\omega)=A_s$ for every $s\in[0,T]$, since $\omega\in\Omega_0$. Subtracting the dynamics of the generalized mean-field trajectory pair from the martingale decomposition of $\Sigma$ componentwise,

$$\Sigma^\gamma_t-S^\gamma_t=\big(\Sigma^\gamma_0-S^\gamma_0\big)+\int_{[0,t]}\Big(b^\gamma(\Sigma_s,A_s)-b^\gamma(S_s,A_s)\Big)ds+M^\gamma_t .$$

The integrand, as a map into $\mathbb{R}^l$, has measurable components and is bounded: the first term is measurable and bounded by $2(l-1)B$ on $\Omega_*$ by clause (a) of the martingale decomposition, and the second by the definition of a generalized mean-field trajectory pair, whose drift is that of the projected extension, which coincides with $b$ on $\Delta^l$ by claim 6 of the lemma on affine-controlled data. Hence the norm bound for vector-valued integrals and the triangle inequality give

$$|\Sigma_t-S_t|\le|\Sigma_0-S_0|+\int_{[0,t]}\big|b(\Sigma_s,A_s)-b(S_s,A_s)\big|\,ds+|M_t| .$$

By the state-Lipschitz bound of the lemma on affine-controlled data, valid since $\Sigma_s$ and $S_s$ lie in $\Delta^l$,

$$\big|b(\Sigma_s,A_s)-b(S_s,A_s)\big|\le\Lambda_b\,|\Sigma_s-S_s|\le\Lambda_b\,\Psi_s ,$$

so by monotonicity of the integral and $|M_t|\le\overline{M}(\omega)$,

$$|\Sigma_t-S_t|\le X+\Lambda_b\int_{[0,t]}\Psi_s\,ds,\qquad X=|\Sigma_0(\omega)-S_0|+\overline{M}(\omega) .$$

The right-hand side is nondecreasing in $t$, so taking the supremum over the times in $[0,t]$ on the left gives

$$\Psi_t\le X+\Lambda_b\int_{[0,t]}\Psi_s\,ds\qquad(t\in[0,T]).$$

Step 3: Gronwall and conclusion. The function $t\mapsto\Psi_t$ is bounded and measurable, so Gronwall's lemma for bounded measurable functions, applied with $a=X$ and $c=\Lambda_b$, gives $\Psi_t\le X\,e^{\Lambda_bt}$ for every $t\in[0,T]$. At $t=T$,

$$\Psi(\omega)=\Psi_T\le e^{\Lambda_bT}\Big(|\Sigma_0(\omega)-S_0|+\overline{M}(\omega)\Big).$$

Squaring and using $(p+q)^2\le2p^2+2q^2$, valid because $(p-q)^2\ge0$,

$$\Psi(\omega)^2\le2e^{2\Lambda_bT}\Big(|\Sigma_0(\omega)-S_0|^2+\overline{M}(\omega)^2\Big)\qquad\text{for every }\omega\in\Omega_* .$$

Both sides of this inequality are bounded random variables: $\Psi\le2$ and $|\Sigma_0-S_0|\le2$ as in Step 1, while, with $K_M=2+2(l-1)BT$ as in the martingale bound lemma, $\overline{M}\le\sqrt{l}\,K_M$ because each $\overline{M^\gamma}\le K_M$ by the supremum lemma. Since $P(\Omega_*)=1$, taking expectations by the passage of almost sure inequalities between bounded random variables to expectations, using linearity of the integral and then the bound $\mathbb{E}[\overline{M}^{\,2}]\le8l(l-1)BT/N$ of the martingale bound lemma gives

$$\mathbb{E}\big[\Psi^2\big]\le2e^{2\Lambda_bT}\Big(\mathbb{E}\big[|\Sigma_0-S_0|^2\big]+\frac{8l(l-1)BT}{N}\Big). \qquad\blacksquare$$