Proof of Uniform Convergence in Probability of the N-Agent Empirical State Measure to the Unique Optimal Mean-Field Trajectory

Throughout we use the elementary order arithmetic of Elementary Order Arithmetic in an Ordered Field, the properties of the absolute value of Properties of the Absolute Value in an Ordered Field and the properties of the Euclidean norm of Elementary Properties of the Euclidean Norm on $\mathbb{R}^n$. Write $K_{b}$ for the constant of claim 2 of the flow stability lemma, so that every mean-field flow satisfies $|S_{t}(x_{0},\xi)-S_{r}(x_{0},\xi)|\le K_{b}|t-r|$ for all $r,t\in[0,T]$.

Step 1. Proof of claim 1. By claim 1 of the optimal-set structure lemma the set $\mathcal{M}^{*}_{\sigma}$ is nonempty. By the definition of the optimal trajectories the set $\mathcal{S}^{*}_{\sigma}$ consists exactly of the flows $S(\sigma,\zeta)$ with $\zeta\in\mathcal{M}^{*}_{\sigma}$; since $\mathcal{S}^{*}_{\sigma}=\{S^{*}\}$, every such flow equals $S^{*}$. In particular, fixing $\zeta_{0}\in\mathcal{M}^{*}_{\sigma}$ we have $S^{*}=S(\sigma,\zeta_{0})$, so $S^{*}_{t}\in\Delta^{l}$ for every $t\in[0,T]$ and $|S^{*}_{t}-S^{*}_{r}|\le K_{b}|t-r|$ for all $r,t\in[0,T]$, by claim 2 of the flow stability lemma.

Let $x_{0}\in\Delta^{l}$ and $\xi\in\mathcal{U}_{\mathcal{A}}$. By claim 2 of the optimal-set structure lemma the supremum

exists for every $\zeta\in\mathcal{U}_{\mathcal{A}}$, and there is $\zeta^{\dagger}\in\mathcal{M}^{*}_{\sigma}$ with $D(x_{0},\xi)=\Psi(x_{0},\xi,\zeta^{\dagger})$. By the first part $S(\sigma,\zeta^{\dagger})=S^{*}$, so

and in particular this supremum exists. This proves claim 1.

Step 2. Proof of claim 2. Fix natural numbers $N$ and $n$.

Measurability of the sections. Let $t\in[0,T]$. By the definition of a solution of the controlled $N$-agent dynamics the state processes are families of random variables, so $\sigma^{i}_{t}$ is measurable for every $i$; hence the occupation indicator $\eta^{i,\gamma}_{t}$, which is the function equal to $1$ on the event where $\sigma^{i}_{t}$ takes the value $\gamma$ and to $0$ elsewhere, is a random variable, and so is the component $\Sigma^{N,\gamma}_{t}$, being the sum of the $N$ indicators $\eta^{i,\gamma}_{t}$ divided by $N$: this follows from the lemma on sequentially continuous functions of measurable Euclidean maps, applied to the sequentially continuous map that sends a point of $\mathbb{R}^{N}$ to the sum of its coordinates divided by $N$. Also, as recorded in that definition, $\Sigma^{N}_{t}(\omega)$ lies in $\Delta^{l}$ for every $\omega$, because each agent occupies exactly one state. Applying the same lemma to the sequentially continuous map sending $x\in\Delta^{l}$ to $|x-S^{*}_{t}|$ shows that $\omega\mapsto|\Sigma^{N}_{t}(\omega)-S^{*}_{t}|$ is a random variable.

The maps $g^{N}_{n}$. The set $Q_{n}$ is finite and nonempty, and $\mathbf{1}_{\Omega^{N}_{*}}$ is a random variable because $\Omega^{N}_{*}\in\mathcal{F}^{N}$. The map $g^{N}_{n}$ is obtained from the finitely many random variables $\omega\mapsto|\Sigma^{N}_{t}(\omega)-S^{*}_{t}|$ with $t\in Q_{n}$, together with $\mathbf{1}_{\Omega^{N}_{*}}$, by forming the maximum of the first group and multiplying by the last; both operations are given by sequentially continuous maps on the corresponding Euclidean space, so $g^{N}_{n}$ is a random variable by the same lemma. For every $t\in[0,T]$ and every $\omega$, both $\Sigma^{N}_{t}(\omega)$ and $S^{*}_{t}$ lie in $\Delta^{l}$ and therefore have Euclidean norm at most $1$ by claim 1 of the compactness lemma, so claims 5 and 6 of the norm lemma give $|\Sigma^{N}_{t}(\omega)-S^{*}_{t}|\le2$. Hence $0\le g^{N}_{n}(\omega)\le2$ for every $\omega$.

The map $W_{N}$. For every $\omega$ the set $\{g^{N}_{n}(\omega):n\in\mathbb{N}\}$ is nonempty and bounded above by $2$, so its least upper bound exists by Least Upper Bound Property of the Real Numbers; thus $W_{N}(\omega)$ is well defined, and $0\le g^{N}_{1}(\omega)\le W_{N}(\omega)\le2$, the last inequality because $2$ is an upper bound and the supremum is the least one. The family $(g^{N}_{n})_{n\in\mathbb{N}}$ consists of random variables bounded in absolute value by $2$, so $W_{N}$ is a random variable by claim 1 of the measurable-limits toolkit.

Identification on $\Omega^{N}_{*}$. Let $\omega\in\Omega^{N}_{*}$ and put $\Phi(t)=|\Sigma^{N}_{t}(\omega)-S^{*}_{t}|$ for $t\in[0,T]$; then $0\le\Phi\le2$ and, since $\mathbf{1}_{\Omega^{N}_{*}}(\omega)=1$, we have $g^{N}_{n}(\omega)=\max\{\Phi(t):t\in Q_{n}\}$ for every $n$. The set $\{\Phi(t):t\in[0,T]\}$ is nonempty and bounded above by $2$, so its least upper bound $\Phi^{*}$ exists.

Each $g^{N}_{n}(\omega)$ equals $\Phi(t)$ for some $t\in Q_{n}\subseteq[0,T]$, hence $g^{N}_{n}(\omega)\le\Phi^{*}$; as $\Phi^{*}$ is then an upper bound of the set whose least upper bound is $W_{N}(\omega)$, we get $W_{N}(\omega)\le\Phi^{*}$.

Conversely let $t\in[0,T]$. If $t=T$ then $t\in Q_{1}$, so $\Phi(t)\le g^{N}_{1}(\omega)\le W_{N}(\omega)$. Suppose $t<T$. For each $n$ the set of elements of $Q_{n}$ that are at least $t$ is nonempty, since it contains $T$, and finite, so it has a least element $t_{n}$. If $t_{n}=0$ then $t\le0$, so $t=0=t_{n}$; otherwise $t_{n}=(j/2^{n})T$ with $j\in\{1,\dots,2^{n}\}$, the number $((j-1)/2^{n})T$ also lies in $Q_{n}$ and is smaller than $t$ by minimality of $t_{n}$, whence $t_{n}-t<t_{n}-((j-1)/2^{n})T=T/2^{n}$. In either case $0\le t_{n}-t\le T/2^{n}$. An easy induction gives $n\le2^{n}$ for every natural number $n$, so $T/2^{n}\le T/n$; the sequence $(1/n)_{n\in\mathbb{N}}$ converges to $0$ by claim 3 of the Archimedean property of the real numbers, hence so does $(T/n)_{n\in\mathbb{N}}$ by claim 3 of the arithmetic of limits of real sequences, and therefore $(t_{n})_{n\in\mathbb{N}}$ converges to $t$ by claim 3 of the order properties of limits of real sequences.

Since $t\in[0,T)$ and $\omega\in\Omega^{N}_{*}$, the path $s\mapsto\Sigma^{N,\gamma}_{s}(\omega)$ is right-continuous at $t$ for every $\gamma$, by the martingale bound for the empirical state measure; as $t\le t_{n}$ for every $n$ and $(t_{n})$ converges to $t$, the real sequence $(\Sigma^{N,\gamma}_{t_{n}}(\omega))_{n\in\mathbb{N}}$ converges to $\Sigma^{N,\gamma}_{t}(\omega)$ for every $\gamma$. Writing $x^{(n)}=\Sigma^{N}_{t_{n}}(\omega)-\Sigma^{N}_{t}(\omega)$, claim 1 of the norm lemma together with the definition of the dot product gives that $|x^{(n)}|^{2}$ is the sum of the squares of the coordinates of $x^{(n)}$, each of which converges to $0$; so $(|x^{(n)}|^{2})_{n\in\mathbb{N}}$ converges to $0$ by claims 1 and 2 of the arithmetic of limits. Consequently $(|x^{(n)}|)_{n\in\mathbb{N}}$ converges to $0$: given a real $\varepsilon>0$, we have $|x^{(n)}|^{2}<\varepsilon^{2}$ for all large $n$, while $\varepsilon\le|x^{(n)}|$ would give $\varepsilon^{2}\le|x^{(n)}|^{2}$ by two applications of claim 10 of the order-arithmetic lemma in the nonstrict form obtained by adjoining the case of equality. Finally $|S^{*}_{t_{n}}-S^{*}_{t}|\le K_{b}|t_{n}-t|$, and the right-hand side converges to $0$, so $(|S^{*}_{t_{n}}-S^{*}_{t}|)_{n\in\mathbb{N}}$ converges to $0$ by claim 3 of the order properties of limits.

Now, for every $n$, two applications of claim 6 of the norm lemma, together with claim 5 there, give

where $\beta_{n}=|x^{(n)}|+|S^{*}_{t_{n}}-S^{*}_{t}|$ and where the middle term was bounded by $W_{N}(\omega)$ because $t_{n}\in Q_{n}$ gives $\Phi(t_{n})\le g^{N}_{n}(\omega)\le W_{N}(\omega)$. The sequence $(\beta_{n})_{n\in\mathbb{N}}$ converges to $0$ by claim 1 of the arithmetic of limits. If $W_{N}(\omega)<\Phi(t)$, then $c=\Phi(t)-W_{N}(\omega)$ would be positive and $c\le\beta_{n}$ for every $n$, contradicting the convergence of $(\beta_{n})$ to $0$. Hence $\Phi(t)\le W_{N}(\omega)$ for every $t\in[0,T]$, so $W_{N}(\omega)$ is an upper bound of $\{\Phi(t):t\in[0,T]\}$ and therefore $\Phi^{*}\le W_{N}(\omega)$. Combining the two inequalities, $W_{N}(\omega)=\Phi^{*}$, which is the last assertion of claim 2.

Step 3. Proof of claim 3. Fix a real $\varepsilon>0$ and a natural number $N$.

A pathwise bound. Let $\omega\in\Omega^{N}_{*}$ and write $S^{\omega}=S(\Sigma^{N}_{0}(\omega),\hat{\alpha}^{N}(\omega))$, which is defined because $\Sigma^{N}_{0}(\omega)\in\Delta^{l}$ and $\hat{\alpha}^{N}(\omega)\in\mathcal{U}_{\mathcal{A}}$ by claim 2 of the realized-control lemma. For every $t\in[0,T]$, claim 6 of the norm lemma, claim 1 of the comparison lemma for the $N$-agent system and the mean-field flow, and claim 1 above give

the last term because by claim 1 the number $D_{N}(\omega)=D(\Sigma^{N}_{0}(\omega),\hat{\alpha}^{N}(\omega))$ is the supremum of the numbers $|S^{\omega}_{t}-S^{*}_{t}|$ over $t\in[0,T]$, hence an upper bound of each of them. Taking the least upper bound over $t\in[0,T]$ and using claim 2 we get

Splitting the event. Put $F_{N}=\{\omega\in\Omega^{N}:\varepsilon/2\le e^{\Lambda_{b}T}\overline{M}^{N}(\omega)\}$ and $G_{N}=\{\omega\in\Omega^{N}:\varepsilon/2\le D_{N}(\omega)\}$, which are events because $\overline{M}^{N}$ is a random variable by the martingale bound and $D_{N}$ is one by claim 4 of the convergence theorem for the optimal $N$-agent value. If $\omega\in\Omega^{N}_{*}$ satisfies $\varepsilon\le W_{N}(\omega)$ but lies in neither $F_{N}$ nor $G_{N}$, then adding the two strict inequalities by claim 3 of the order-arithmetic lemma and using claim 8 there, which gives $\varepsilon/2+\varepsilon/2=\varepsilon$, yields $e^{\Lambda_{b}T}\overline{M}^{N}(\omega)+D_{N}(\omega)<\varepsilon\le W_{N}(\omega)$, contradicting the pathwise bound. Hence

Since $P^{N}(\Omega^{N}_{*})=1$, claim 3 of the basic properties of a measure gives $P^{N}(\Omega^{N}\setminus\Omega^{N}_{*})=0$; so claim 2 (monotonicity) and claim 4 (countable subadditivity) there, the latter applied to the sequence whose first three terms are $F_{N}$, $G_{N}$ and $\Omega^{N}\setminus\Omega^{N}_{*}$ and all of whose further terms are empty, give

Estimating the martingale term. Put $a=\varepsilon^{2}/(4e^{2\Lambda_{b}T})$, a positive real number. If $\omega\in F_{N}$ then, multiplying by the positive number $e^{-\Lambda_{b}T}$ (claim 7 of the order-arithmetic lemma for its positivity and claim 10 in the nonstrict form for the multiplication), $\varepsilon/(2e^{\Lambda_{b}T})\le\overline{M}^{N}(\omega)$, and two further applications of claim 10, both sides being nonnegative, give $a\le(\overline{M}^{N}(\omega))^{2}$. Hence $F_{N}$ is contained in the event where $a\le(\overline{M}^{N})^{2}$, and monotonicity together with Markov's inequality of the lemma on Markov's and Chebyshev's inequalities, applied to the nonnegative random variable $(\overline{M}^{N})^{2}$ and the positive number $a$, gives

Together with the previous display this proves the inequality asserted in claim 3.

Passing to the limit. The sequence $(1/N)_{N\in\mathbb{N}}$ converges to $0$ by claim 3 of the Archimedean property, so the sequence with $N$-th term $32\,l\,(l-1)\,B\,T\,e^{2\Lambda_{b}T}/(N\varepsilon^{2})$ converges to $0$ by claim 3 of the arithmetic of limits. The sequence $(P^{N}(G_{N}))_{N\in\mathbb{N}}$ converges to $0$ by claim 4 of the convergence theorem for the optimal $N$-agent value, applied with the positive real number $\varepsilon/2$. Their sum converges to $0$ by claim 1 of the arithmetic of limits, and since $0\le P^{N}(\{\varepsilon\le W_{N}\})$ is bounded above by that sum for every $N$, claim 3 of the order properties of limits gives that $(P^{N}(\{\varepsilon\le W_{N}\}))_{N\in\mathbb{N}}$ converges to $0$. This completes the proof of claim 3 and of the corollary.