Proof of Mean-Square Tracking of the Mean-Field Trajectory under an Open-Loop Control

Write $K_b=2\sqrt{l}(l-1)B$ and let $\Omega_*$, $\overline{M^\gamma}$ and $\overline{M}$ be as in the martingale bound for the empirical state measure, so that $P(\Omega_*)=1$, $\Omega_*\subseteq\Omega_0$ with $\Omega_0$ the regular event, and $|M_t(\omega)|\le\overline{M}(\omega)$ for every $t\in[0,T]$ and $\omega\in\Omega_*$.

Step 1: $\Psi$ is a random variable. Both $\Sigma_t$ and $S_t$ lie in the probability simplex, so $|\Sigma_t|\le1$ and $|S_t|\le1$, whence $|\Sigma_t-S_t|\le2$ for every $t$ and every $\omega$, by the triangle inequality. For $\omega\in\Omega_*$ each path $t\mapsto\Sigma^\gamma_t(\omega)$ is right-continuous at every $t\in[0,T)$ by the martingale bound for the empirical state measure, hence so is $t\mapsto\Sigma_t(\omega)$, and $t\mapsto S_t$ is continuous; hence $t\mapsto\Sigma_t(\omega)-S_t$ is right-continuous there, and so is $t\mapsto|\Sigma_t(\omega)-S_t|$. Applying the supremum lemma for bounded right-continuous processes to the family of random variables $Z_t=|\Sigma_t-S_t|$ with $K=2$ and the event $\Omega_*$, the map $\Psi$ is a random variable with $0\le\Psi\le2$, and $\Psi(\omega)=\sup_{t\in[0,T]}|\Sigma_t(\omega)-S_t|$ for every $\omega\in\Omega_*$.

Step 2: a pathwise integral inequality. Fix $\omega\in\Omega_*$ and write $\Psi_t=\sup_{s\in[0,t]}|\Sigma_s(\omega)-S_s|$ for $t\in[0,T]$, so that $\Psi_T=\Psi(\omega)$ by Step 1. The function $t\mapsto\Psi_t$ is nondecreasing with values in $[0,2]$, hence bounded and measurable by measurability of monotone functions.

By the open-loop policy lemma, $\alpha_s(\omega)=A_s$ for every $s\in[0,T]$, since $\omega\in\Omega_0$. Subtracting the dynamics of the generalized mean-field trajectory pair from the martingale decomposition of $\Sigma$ componentwise,

The integrand, as a map into $\mathbb{R}^l$, has measurable components and is bounded: the first term is measurable and bounded by $2(l-1)B$ on $\Omega_*$ by clause (a) of the martingale decomposition, and the second by the definition of a generalized mean-field trajectory pair. Hence the norm bound for vector-valued integrals and the triangle inequality give

By the state-Lipschitz bound of the projected-extension lemma, valid since $\Sigma_s$ and $S_s$ lie in $\Delta^l$,

so by monotonicity of the integral and $|M_t|\le\overline{M}(\omega)$,

The right-hand side is nondecreasing in $t$, so taking the supremum over the times in $[0,t]$ on the left gives

Step 3: Gronwall and conclusion. The function $t\mapsto\Psi_t$ is bounded and measurable, so Gronwall's lemma for bounded measurable functions, applied with $a=X$ and $c=\Lambda_b$, gives $\Psi_t\le X\,e^{\Lambda_bt}$ for every $t\in[0,T]$. At $t=T$,

Squaring and using $(p+q)^2\le2p^2+2q^2$, valid because $(p-q)^2\ge0$,

Both sides of this inequality are bounded random variables: $\Psi\le2$ and $|\Sigma_0-S_0|\le2$ as in Step 1, while $\overline{M}\le\sqrt{l}\,K_M$ because each $\overline{M^\gamma}\le K_M$ by the supremum lemma. Since $P(\Omega_*)=1$, taking expectations by the passage of almost sure inequalities between bounded random variables to expectations, using linearity of the integral and then the bound $\mathbb{E}[\overline{M}^{\,2}]\le8l(l-1)BT/N$ of the martingale bound lemma gives