Proof of Martingale Decomposition of the Empirical State Measure and the Observation Process

Throughout, fix a solution and work with its occupation indicators $\eta^{i,\gamma}_t$, counters $N^{i,\sigma\gamma}_t$, $\tilde{N}^{i,\upsilon}_t$, consumed clock times $A^{i,\sigma\gamma}_t$, $\tilde{A}^{i,\upsilon}_t$, regular event $\Omega_0$, and system filtration, all in the sense of Solution of the Controlled N-Agent Dynamics; pathwise identities below are asserted on $\Omega_0$, where conditions 1--6 of that definition hold. Write $M^a_t=N^a_t-A^a_t$ for the compensated counters, indexed by clock labels $a$ as in that lemma.

Step 1: part (a). Each path $s\mapsto\eta^{i,\gamma}_s$ is piecewise constant with finitely many pieces by condition 1 of Solution of the Controlled N-Agent Dynamics, hence measurable on $[0,T]$ with the trace Borel $\sigma$-algebra, and therefore so is each component $s\mapsto\Sigma^\gamma_s$, a finite sum of indicators divided by $N$; measurability of finite sums follows from Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable applied with $g$ the (continuous) sum of coordinates and $f$ the vector of indicator paths. Similarly, by condition 5 of Solution of the Controlled N-Agent Dynamics, on each of the finitely many intervals between observation events (finitely many by condition 3, since the observation total agrees with a counting path on $[0,T]$) the control path $s\mapsto\alpha_s$ agrees with a section $s\mapsto h^j_k(s,\tau^\circ,v^\circ)$ of a policy function with the record entries frozen; such a section is measurable because $s\mapsto(s,\tau^\circ)$ maps $[0,T]$ into $[0,T]\times R_k(T)$ and pulls each relatively open set back to a relatively open subset of $[0,T]$, hence pulls the generated $\sigma$-algebra into the trace Borel $\sigma$-algebra. So each component path $s\mapsto\alpha^j_s$ is measurable on $[0,T]$ (a finite patching of measurable functions on subintervals). Now each of the functions $(\Sigma,\alpha)\mapsto b^\gamma(\Sigma,\alpha)$, $\Sigma\mapsto\tilde{b}^\upsilon(\Sigma)$, and $(\Sigma,\alpha)\mapsto\Theta^{\gamma\delta}(\Sigma,\alpha)$ is a finite sum of products of coordinate maps and members of the transition-rate family or observation-rate family, hence sequentially continuous on $\Delta^l\times\mathbb{R}^m$ (respectively $\Delta^l$) by the continuity conditions of those definitions together with the fact that limits of products of convergent real sequences are the products of the limits; therefore Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable applies to the measurable path $s\mapsto(\Sigma_s,\alpha_s)$ and yields the asserted path measurability of $s\mapsto b^\gamma(\Sigma_s,\alpha_s)$, $s\mapsto\tilde{b}^\upsilon(\Sigma_s)$, $s\mapsto\Theta^{\gamma\delta}(\Sigma_s,\alpha_s)$ on $\Omega_0$.

For the bounds, recall from Transition-Rate Family that $0\le\beta\le B$, from Observation-Rate Family that $0\le\tilde{\beta}\le\tilde{B}$, and from Probability Simplex that $\Sigma^\gamma_s\ge0$ and $\sum_\gamma\Sigma^\gamma_s=1$. From the formula in Aggregate State Drift, $|b^\gamma|\le B\sum_{\sigma\neq\gamma}\Sigma^\sigma+(l-1)B\Sigma^\gamma\le B+(l-1)B\le 2(l-1)B$ since $l\ge2$. From Aggregate Fluctuation Covariance, the diagonal entries satisfy $|\Theta^{\gamma\gamma}|\le B\sum_{\sigma\neq\gamma}\Sigma^\sigma+(l-1)B\Sigma^\gamma\le 2(l-1)B$ likewise, and the off-diagonal entries satisfy $|\Theta^{\gamma\delta}|\le B(\Sigma^\gamma+\Sigma^\delta)\le 2B\le 2(l-1)B$. From Aggregate Observation Drift, $|\tilde{b}^\upsilon|\le\tilde{B}\sum_\sigma\Sigma^\sigma=\tilde{B}$. Bounded measurable paths are Lebesgue integrable on $[0,t]$ by Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval, so all the integrals exist. This proves (a).

Step 2: the aggregate martingales are combinations of compensated counters. Fix $\gamma$ and work on $\Omega_0$. Summing condition 6 of Solution of the Controlled N-Agent Dynamics over $i$ and dividing by $N$,

On the other hand, for fixed $\sigma\neq\gamma'$, the identity $\frac{1}{N}\sum_i\eta^{i,\sigma}_s\,\beta(\sigma,\gamma',\Sigma_s,\alpha_s)=\Sigma^\sigma_s\,\beta(\sigma,\gamma',\Sigma_s,\alpha_s)$ holds pointwise in $s$ on $\Omega_0$, so by linearity of the Lebesgue integral and the definition of the consumed clock times in Solution of the Controlled N-Agent Dynamics,

Combining with the formula for $b^\gamma$ in Aggregate State Drift and subtracting, on $\Omega_0$,

that is, $M^\gamma_t=\frac{1}{N}\sum_a c^\gamma_a M^a_t$ where the sum runs over transition clock labels $a=(i,\sigma\gamma')$ and $c^\gamma_{(i,\sigma\gamma')}=\mathbf{1}_{\{\gamma'=\gamma\}}-\mathbf{1}_{\{\sigma=\gamma\}}\in\{-1,0,1\}$ (we write $\mathbf{1}_{\{\cdot\}}$ for the indicator equal to $1$ when the subscripted condition holds and $0$ otherwise). Entirely analogously, condition 4 of Solution of the Controlled N-Agent Dynamics, the identity $\frac{1}{N}\sum_i\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\sum_\sigma\Sigma^\sigma_s\tilde{\beta}(\sigma,\upsilon,\Sigma_s)=\tilde{b}^\upsilon(\Sigma_s)$ (valid on $\Omega_0$ since $\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\sum_\sigma\eta^{i,\sigma}_s\tilde{\beta}(\sigma,\upsilon,\Sigma_s)$), and Aggregate Observation Drift give, on $\Omega_0$,

where $M^{(i,\upsilon)}_t=\tilde{N}^{i,\upsilon}_t-\tilde{A}^{i,\upsilon}_t$ is the compensated counter of the observation clock label $(i,\upsilon)$.

Step 3: part (b). By part (a) of the compensated counter lemma, each $M^a$ is a square-integrable martingale with respect to the system filtration with $M^a_0=0$, and a finite linear combination of square-integrable martingales with real coefficients is again one: adaptedness and square-integrability are preserved under finite linear combinations (for square-integrability use the triangle inequality for the mean-square norm), and the martingale property in the averaged form of Square-Integrable Martingale, Submartingale, and Supermartingale is linear in the process. The Step 2 identities hold only on $\Omega_0$, so what they show is that $M^\gamma_t$ (and likewise $\tilde{M}^\upsilon_t$) is almost surely equal to the corresponding combination $Z_t=\frac1N\sum_a c^\gamma_aM^a_t$. Square-integrability and the averaged martingale identity transfer under almost sure equality, since expectations of the involved products are unchanged. Adaptedness also transfers: $M^\gamma_t$ is $\mathcal{F}$-measurable (a difference of a random variable and an integral that is measurable in $\omega$, by the same section-and-Tonelli argument recorded in condition 2 of Solution of the Controlled N-Agent Dynamics applied to the bounded jointly measurable integrand $\mathbf{1}_{\Omega_0}b^\gamma(\Sigma_s,\alpha_s)$, whose joint measurability follows from condition 2 and Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable as in Step 1), and for every Borel set $S$ the events $\{M^\gamma_t\in S\}$ and $\{Z_t\in S\}$ differ only within the complement of $\Omega_0$; both difference sets are events of probability zero, and by Solution of the Controlled N-Agent Dynamics the system filtration contains every event of probability zero, so $\{M^\gamma_t\in S\}\in\mathcal{F}^{\mathrm{sys}}_t$. Hence each $M^\gamma$ and each $\tilde{M}^\upsilon$ is a square-integrable martingale, and $M^\gamma_0=\tilde{M}^\upsilon_0=0$ almost surely from Step 2 with $t=0$, indeed surely after noting both sides vanish identically off $\Omega_0$... more precisely $M^\gamma_0=\Sigma^\gamma_0-\Sigma^\gamma_0-0=0$ everywhere and $\tilde{M}^\upsilon_0=\Upsilon^\upsilon_0$, which vanishes on $\Omega_0$ by conditions 3--4 (the counters vanish at $t=0$ because the consumed clock times do and clock paths are counting paths). This proves (b).

Step 4: part (c). Fix $0\le r\le t\le T$ and $D\in\mathcal{F}^{\mathrm{sys}}_r$. All the products below are integrable by the integrability assertion of the compensated counter lemma and finite summation, and almost sure equality (Step 3) lets us compute with $Z_t=\frac1N\sum_a c^\gamma_aM^a_t$ in place of $M^\gamma_t$. By Step 2 and bilinearity,

Applying part (b) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales to every pair $(a,b)$ and re-assembling the $r$-terms by the same bilinearity,

It remains to identify the last sum. For a transition clock label $a=(i,\sigma\gamma')$ we have, as in Step 2, $\sum_i (A^{i,\sigma\gamma'}_t-A^{i,\sigma\gamma'}_r)=N\int_{[r,t]}\Sigma^\sigma_s\beta(\sigma,\gamma',\Sigma_s,\alpha_s)\,ds$ on $\Omega_0$ (additivity of the Lebesgue integral over $[0,r]$ and $[r,t]$ from Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval). Hence, on $\Omega_0$,

For $\gamma=\delta$, the summand weight $(\mathbf{1}_{\{\gamma'=\gamma\}}-\mathbf{1}_{\{\sigma=\gamma\}})^2$ equals $1$ exactly when $\gamma'=\gamma$ or $\sigma=\gamma$ (never both, as $\sigma\neq\gamma'$) and $0$ otherwise, so the inner sum equals $\sum_{\sigma\neq\gamma}\Sigma^\sigma\beta(\sigma,\gamma,\Sigma_s,\alpha_s)+\sum_{\gamma'\neq\gamma}\Sigma^\gamma\beta(\gamma,\gamma',\Sigma_s,\alpha_s)=\Theta^{\gamma\gamma}(\Sigma_s,\alpha_s)$ by Aggregate Fluctuation Covariance. For $\gamma\neq\delta$, the weight is nonzero only for $(\sigma,\gamma')=(\gamma,\delta)$, where it is $(\mathbf{1}_{\{\delta=\gamma\}}-\mathbf{1}_{\{\gamma=\gamma\}})(\mathbf{1}_{\{\delta=\delta\}}-\mathbf{1}_{\{\gamma=\delta\}})=(-1)(1)=-1$, and for $(\sigma,\gamma')=(\delta,\gamma)$, where it is likewise $-1$; all other pairs give $0$. The inner sum is then $-\Sigma^\gamma\beta(\gamma,\delta,\Sigma_s,\alpha_s)-\Sigma^\delta\beta(\delta,\gamma,\Sigma_s,\alpha_s)=\Theta^{\gamma\delta}(\Sigma_s,\alpha_s)$ by Aggregate Fluctuation Covariance. Taking expectations of the almost sure pathwise identity and combining with the previous display proves the first identity of (c).

For the observation covariations, Step 2 gives $\tilde{M}^\upsilon=\frac{1}{N}\sum_i M^{(i,\upsilon)}$ almost surely, and part (b) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales leaves only the diagonal pairs $(i,\upsilon)=(i',\upsilon')$, which force $\upsilon=\upsilon'$: when $\upsilon\neq\upsilon'$ no diagonal pairs occur and the $r$-identity holds with nothing added, while for $\upsilon=\upsilon'$ the added term is $\frac{1}{N^2}\sum_i\mathbb{E}[\mathbf{1}_D(\tilde{A}^{i,\upsilon}_t-\tilde{A}^{i,\upsilon}_r)]=\frac{1}{N}\mathbb{E}\big[\mathbf{1}_D\int_{[r,t]}\tilde{b}^\upsilon(\Sigma_s)\,ds\big]$, using $\frac{1}{N}\sum_i\tilde{\beta}(\sigma^i_s,\upsilon,\Sigma_s)=\tilde{b}^\upsilon(\Sigma_s)$ on $\Omega_0$ as in Step 2 and integral additivity from Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval. Finally, a state martingale $M^\gamma$ and an observation martingale $\tilde{M}^\upsilon$ are combinations over disjoint sets of clock labels (transition clocks versus observation clocks), so part (b) of Compensated Counters of the Controlled N-Agent Dynamics are Square-Integrable Martingales contributes no diagonal terms at all and the mixed identity holds with nothing added. This proves (c). $\blacksquare$