Throughout, fix r∈R and write S for the solution furnished by clause (d) of Measurable Reconstruction of the Controlled N-Agent Dynamics from Observation Records: its state processes are the σr,i, its control process is αs=ar(s), its regular event is Ωr with P(Ωr)=1, and {(r,ω):ω∈Ωr}⊆G. Its clocks are the transition and observation clocks of the driving system. We write Xti=σtr,i for the i-th coordinate of Xt. Measurability of real-valued maps on [0,T] or on Ω is always with respect to B[0,T] or F and the Borel σ-algebra of the real line, and Arithmetic, Absolute Values, and Pointwise Limits of Measurable Real-Valued Functions is used for constants, indicators, sums and products of measurable maps.
Step 0 (Σ(x) lies in the lattice). For x∈E the counts #{i:xi=γ}, γ∈{1,…,l}, are nonnegative integers with sum N, each index i having exactly one value xi. Hence the coordinates of Σ(x) are nonnegative with sum 1, so Σ(x) lies in the probability simplex Δl, and NΣ(x)γ is 0 or a natural number; thus Σ(x)∈GN by the definition of the aggregate lattice. In particular GN is nonempty, and it is finite, being contained in the set of points of Rl all of whose coordinates lie in {0,N1,…,1}, which has (N+1)l elements; E has lN≥1 elements and A has Nl(l−1)≥1 elements, so the finiteness and nonemptiness hypotheses of Forward Equation for a Finite-State Jump System Driven by Poisson Clocks with the Fresh-Start Property and Uniqueness for the Forward Equation of a Bounded Jump-Rate Family on a Finite Set hold.
Step 1 (Identities). By the derived notation of Solution of the Controlled N-Agent Dynamics, the occupation indicators of S are ηti,γ=1{Xti=γ} and its empirical state measure is the point with coordinates N1∑i1{Xti=γ}=Σ(Xt)γ, that is, Σ^tr, at every ω∈Ω. Now let ω∈Ωr. Then (r,ω)∈G, so clause (c) of Measurable Reconstruction of the Controlled N-Agent Dynamics from Observation Records gives, for every i and t, exactly one γ with ηtr,i,γ(ω)=1, the others being 0, and σtr,i(ω) is that γ; hence ηtr,i,γ(ω)=1{σtr,i(ω)=γ} for all γ, and therefore Σtr(ω)=Σ^tr(ω). This proves the first assertion of claim 1. Consequently, by condition 2 of Solution of the Controlled N-Agent Dynamics for S (regular event Ωr, control ar), for every ω∈Ω, every e=(i,(σ,γ))∈A and every t∈[0,T],
Tti,σγ(ω)=∫[0,t]1Ωr(ω)1{Xui(ω)=σ}β(σ,γ,Σ^ur(ω),ar(u))du=∫[0,t]1Ωr(ω)ge(u,Xu(ω))du.(1)
Step 2 (D1). By condition 2 of N-Agent Driving System, each Yi,σγ is a stochastic process, that is, a family of random variables indexed by [0,∞) (the earlier version of that definition invoked in N-Agent Driving System defines the term identically), every path of which is a counting path.
Step 3 (D2). Fix e=(i,(σ,γ)) and x∈E. By clause (d) of Measurable Reconstruction of the Controlled N-Agent Dynamics from Observation Records, ar(u)∈A for every u, and Σ(x)∈Δl by Step 0, so 0≤ge(u,x)≤B by condition 1 of Transition-Rate Family. If xi=σ then ge(⋅,x) is the constant 0, which is measurable. If xi=σ then ge(u,x)=β(σ,γ,y,ar(u)) with y=Σ(x); we show more generally that u↦β(σ,γ,y,ar(u)) is measurable for every y∈Δl. The map α↦β(σ,γ,y,α) is sequentially continuous on A⊆Rm: if ζn→ζ in A in Euclidean distance, then the sequence (y,ζn) satisfies the hypothesis of condition 2 of Transition-Rate Family with constant first entries, whence β(σ,γ,y,ζn)→β(σ,γ,y,ζ). Each component of ar:[0,T]→A is measurable with respect to B[0,T] by The Record-Frozen Control Path and Record-Frozen Policy. Hence u↦β(σ,γ,y,ar(u)) is measurable by Sequentially Continuous Functions of Measurable Euclidean Maps are Measurable, applied on the measurable space ([0,T],B[0,T]) with d=m, the subset A and the map ar. Since ar takes values in A (clause (d) of Measurable Reconstruction of the Controlled N-Agent Dynamics from Observation Records) and its components are B[0,T]-measurable (The Record-Frozen Control Path and Record-Frozen Policy), ar is a control path in the sense of that definition; this is the assertion of claim 1 concerning ar.
Step 4 (D3). The system filtration (Ftsys,r)t∈[0,T] of S is a filtration of sub-σ-algebras of F by clause (vii)(e) of Existence, Uniqueness, and Regularity for the Controlled N-Agent Dynamics, and Ωr∈F has probability one. For t∈[0,T] and x∈E, {Xt=x}=⋂i=1N{σtr,i=xi}, and each {σtr,i=xi} lies in Ftsys,r because σr,i is adapted to the system filtration by clause (iv) of Existence, Uniqueness, and Regularity for the Controlled N-Agent Dynamics; so {Xt=x}∈Ftsys,r.
For the product measurability, fix x∈E. By Step 1, for every (u,ω)∈[0,T]×Ω,
1Ωr(ω)1{Xu(ω)=x}=1Ωr(ω)i=1∏Nηur,i,xi(ω),
since on Ωr one has 1{Xui=xi}=ηur,i,xi for every i, and both sides vanish off Ωr. The first factor is the indicator of the measurable rectangle [0,T]×Ωr and is therefore B[0,T]⊗F-measurable. For the remaining factors we show that for all i,γ the map (u,ω)↦ηur,i,γ(ω) is B[0,T]⊗T-measurable; this suffices, because T⊆F makes every measurable rectangle of B[0,T]⊗T a measurable rectangle of B[0,T]⊗F, so that B[0,T]⊗T⊆B[0,T]⊗F by the definition of the product σ-algebra as a generated σ-algebra, and a finite product of measurable real-valued maps is measurable. Consider the map ι:[0,T]×Ω→(R×[0,T])×Ω, ι(u,ω)=((r,u),ω). The family of subsets S of (R×[0,T])×Ω with ι−1(S)∈B[0,T]⊗T is a σ-algebra (preimages commute with complements and countable unions), so ι is measurable from B[0,T]⊗T to (R⊗B[0,T])⊗T as soon as ι−1(A×C)∈B[0,T]⊗T for every A∈R⊗B[0,T] and C∈T. Now ι−1(A×C)=Ar×C with Ar={u∈[0,T]:(r,u)∈A}. The family S of subsets A of R×[0,T] with Ar∈B[0,T] is a σ-algebra, since (R×[0,T]∖A)r=[0,T]∖Ar, (⋃nAn)r=⋃n(An)r, and B[0,T] is a σ-algebra by claim 1 of Restricted Lebesgue Measure and Integral Toolkit on a Compact Interval; and S contains every measurable rectangle A′×B′ with A′∈R and B′∈B[0,T], whose r-section is B′ if r∈A′ and empty otherwise. Hence S⊇R⊗B[0,T], so Ar∈B[0,T] and Ar×C is a measurable rectangle of B[0,T]⊗T. Thus ι is measurable, and (u,ω)↦ηur,i,γ(ω) is the composition of ι with the map (r′,u,ω)↦ηur′,i,γ(ω), which is (R⊗B[0,T])⊗T-measurable by clause (a) of Measurable Reconstruction of the Controlled N-Agent Dynamics from Observation Records; a composition of measurable maps is measurable, the preimage of a set under the composition being the preimage under ι of its preimage under the outer map. The same argument with the map ω↦((r,u),ω) for fixed u (preimages of rectangles A×C are C or empty) shows that ω↦ηur,i,γ(ω) is T-measurable for each fixed u, so that {Σur=y}∈T⊆F for every u∈[0,T] and y∈Rl; this is used in Step 8.
Step 5 (D4). The solution S satisfies conditions 1--4 of Solution of the Controlled N-Agent Dynamics with regular event Ωr, so clause (vii)(b) of Existence, Uniqueness, and Regularity for the Controlled N-Agent Dynamics shows that each Tti,σγ is F-measurable with values in [0,Bt]=[0,Λt]; and by (1), for ω∈Ωr, Tti,σγ(ω)=∫[0,t]ge(u,Xu(ω))du with e=(i,(σ,γ)), as (D4) requires. The counters of Forward Equation for a Finite-State Jump System Driven by Poisson Clocks with the Fresh-Start Property are YTtee=YTti,σγi,σγ=Nti,σγ, the transition counters of condition 3 of Solution of the Controlled N-Agent Dynamics.
Step 6 (H1). Fix ω∈Ωr; conditions 1, 3 and 6 of Solution of the Controlled N-Agent Dynamics hold at ω, and we suppress ω.
Piecewise constancy. By condition 1, for each i there are a count K(i) and times 0<t1(i)<⋯<tK(i)(i)≤T such that Xi is constant on [0,t1(i)), on each [tj(i),tj+1(i)), and on [tK(i)(i),T] (on all of [0,T] if K(i)=0). Let 0<t1<⋯<tK≤T enumerate the union of these finite sets of times (K=0 if the union is empty). Let I be one of the intervals [0,t1), [tk,tk+1) (1≤k<K), [tK,T] (or [0,T] if K=0), with left endpoint ℓ. Fix i and let j be the largest index with tj(i)≤ℓ, or j=0 if there is none. If j<K(i) then tj+1(i)>ℓ, and since tj+1(i) belongs to the union it is at least the next enumerated time after ℓ, which is the right endpoint of I when I=[tk,tk+1) or [0,t1), and does not exist when I=[tK,T] (all enumerated times being ≤tK), so that this case is impossible then. Hence I is contained in the constancy interval of Xi starting at tj(i) (at 0 if j=0), and Xi is constant on I. As this holds for every i, X is constant on I: the path of X is piecewise constant and right-continuous with the times t1,…,tK, as (H1) requires, and Xt− is defined for t∈(0,T].
Jump rule. Let t∈(0,T], and let Nt−e and the notation of (H1) be as in Forward Equation for a Finite-State Jump System Driven by Poisson Clocks with the Fresh-Start Property, and write Nt=∑e∈ANte and Nt− for the grand total and its left limit, called Nt and Nt− there (the bare symbol N denotes the number of agents here); put ΔNte=Nte−Nt−e, a nonnegative integer. By condition 3 each path s↦Nse coincides on [0,T] with a counting path, so it is nondecreasing and integer-valued; the set {Nse:0≤s<t} is a set of integers bounded above by Nte, so its supremum Nt−e is attained at some se<t, and by monotonicity Nse=Nt−e for all s∈[se,t). Choose s′<t with s′≥se for all e∈A and such that Xs=Xt− for s∈(s′,t) (possible by piecewise constancy). Fix s∈(s′,t). Condition 6 at the times t and s, subtracted, gives for all i and γ, using ηi,γ=1{Xi=γ} from Step 1,
1{Xti=γ}−1{Xt−i=γ}=σ=γ∑ΔNti,σγ−γ′=γ∑ΔNti,γγ′.(2)
If Nt=Nt−, then, as Nt−=∑eNt−e, all ΔNte vanish, so (2) gives 1{Xti=γ}=1{Xt−i=γ} for all i,γ, that is, Xt=Xt−. If Nt=Nt−+1, let e0=(i0,(σ0,γ0)) be the unique label with ΔNte0=1, all other ΔNte being 0. For i=i0 the right side of (2) vanishes for every γ, so Xti=Xt−i. For i=i0 and γ=γ0, the right side of (2) equals ΔNti0,σ0γ0=1, no label of the form (i0,(γ0,γ′)) being e0 since γ0=σ0; hence 1{Xti0=γ0}=1+1{Xt−i0=γ0}≥1, forcing Xti0=γ0. Thus Xt=ϕe0(Xt−), as (H1) requires.
Step 7 (H2). Fix s∈[0,T], the time called r in (H2), and put ϱs=B(T−s). The residual clocks of (H2) are Y^ue,s=YTsi,σγ+ui,σγ−YTsi,σγi,σγ for e=(i,(σ,γ)), which are the residual transition clocks Y^ui,σγ of Fresh-Start Property of the Controlled N-Agent Dynamics applied at time s to the solution S; that lemma applies with the policy h^r, which is A-valued by clause (d) of Measurable Reconstruction of the Controlled N-Agent Dynamics from Observation Records, with the solution S on the same driving system, and with its time parameter equal to s. By its clause (a), each residual clock is a homogeneous Poisson process with rate 1: each Y^ue,s is a random variable, and by conditions 2 and 3 of Stochastic Process, Independent Increments, and Inhomogeneous Poisson Process (the earlier version of that definition, which is the one invoked in Fresh-Start Property of the Controlled N-Agent Dynamics, states conditions 1--3 and the notion of a homogeneous Poisson process with rate 1 identically; the two versions differ only in the continuity requirement imposed on a general intensity function, which plays no role here), for all real 0=u0<u1<⋯<up≤ϱs the increments Y^uqe,s−Y^uq−1e,s, 1≤q≤p, are independent, the q-th having the Poisson distribution with parameter uq−uq−1. For the independence requirement, clause (b) of Fresh-Start Property of the Controlled N-Agent Dynamics states that the family consisting of Fssys,r, the σ-algebras σ(Y^ui,σγ:u≥0) and the σ-algebras generated by the residual observation clocks is independent. The family required by (H2) consists of Fssys,r and the σ-algebras Ge=σ(Y^ue,s:0≤u≤ϱs), e∈A; each Ge is contained in σ(Y^ui,σγ:u≥0), its generators in the sense of Sigma-Algebra Generated by Random Variables and Independence of Sigma-Algebras being among those of the latter, so that the smaller generated σ-algebra is contained in the larger. Given finitely many distinct members of the (H2) family and events in them, these events lie in the corresponding distinct members of the fresh-start family, so the product identity of Sigma-Algebra Generated by Random Variables and Independence of Sigma-Algebras holds. Thus (H2) holds.
Step 8 (Claims 1 and 2). Steps 2--7 establish (D1)--(D4), (H1) and (H2), which proves claim 1, so conclusions (a) and (b) of Forward Equation for a Finite-State Jump System Driven by Poisson Clocks with the Fresh-Start Property hold: for every F~:E→R, all 0≤s≤t≤T and D∈Fssys,r,
E[(F~(Xt)−F~(Xs))1D]=E[1D∫[s,t]1Ωre∈A∑ge(u,Xu)(F~(ϕe(Xu))−F~(Xu))du],(3)
the inner integral being defined everywhere and F-measurable; and for s∈[0,T) and D∈Fssys,r the family μuD(x)=P(D∩{Xu=x}), u∈[s,T], is a solution of the forward equation on [s,T] for the rates quE(x,x′)=∑e:ϕe(x)=x′ge(u,x) on E (called q in that lemma).
Generator identity. Let F:GN→R, with F(y+N1vc) read as F(y) when y+N1vc∈/GN. We claim that for all u∈[0,T] and x∈E, writing y=Σ(x),
e∈A∑ge(u,x)(F(Σ(ϕe(x)))−F(y))=c=(σ,γ)∑Nyσβ(σ,γ,y,ar(u))(F(y+N1vc)−F(y))=y′∈GN,y′=y∑qur(y,y′)(F(y′)−F(y)).(4)
For the first equality fix c=(σ,γ) and consider the labels e=(i,c). If xi=σ then ge(u,x)=0. If xi=σ, then ϕe(x) differs from x only in coordinate i, which changes from σ to γ, so the count of σ decreases by one and that of γ increases by one: Σ(ϕe(x))=y+N1(δγ−δσ)=y+N1vc, a point of GN by Step 0, and ge(u,x)=β(σ,γ,y,ar(u)). There are Nyσ indices i with xi=σ, so the labels (i,c) contribute Nyσβ(σ,γ,y,ar(u))(F(y+N1vc)−F(y)) to the left side, which is also the c-term of the middle expression; when yσ=0 both are 0 whatever the reading of F(y+N1vc). For the second equality: if yσ≥N1 then y+N1vc has nonnegative coordinates yσ−N1, yγ+N1 and yγ′ (γ′∈/{σ,γ}), all multiples of N1 with sum 1, so it lies in GN, whereas if yσ=0 its σ-coordinate is negative and it does not; distinct labels give distinct points y+N1vc, as vc determines c. Hence, by the definition of qr, the right side of (4) is the sum over the labels c with yσ≥N1 of Nyσβ(σ,γ,y,ar(u))(F(y+N1vc)−F(y)), and the remaining terms of the middle expression vanish.
The displayed identity of claim 2. Apply (3) with F~=F∘Σ, so that F~(Xt)=F(Σ^tr), and use the first equality of (4) at x=Xu(ω), where y=Σ^ur(ω): the integrand of (3) becomes the integrand displayed in claim 2, as a function of (u,ω). The measurability assertion for the inner integral is that of conclusion (a) of Forward Equation for a Finite-State Jump System Driven by Poisson Clocks with the Fresh-Start Property, the two integrands being the same function.
The rates qr. For y=y′ in GN the map u↦qur(y,y′) is either the constant 0 or u↦Nyσβ(σ,γ,y,ar(u)), a scalar multiple of a map shown measurable in Step 3; so it is measurable. Since there are l(l−1) transition labels, 0≤yσ≤1 for y∈GN⊆Δl and β≤B, ∑y′=yqur(y,y′)≤∑cNyσB≤l(l−1)NB.
νD solves the forward equation. Fix s∈[0,T) and D∈Fssys,r. For u∈[s,T] and y∈GN, the events {Xu=x}, x∈E, partition Ω and {Σ^ur=y} is the union of those with Σ(x)=y, so by additivity of P
νuD(y)=P(D∩{Σ^ur=y})=x∈E:Σ(x)=y∑μuD(x);
moreover {Σur=y} is an event (Step 4) that differs from {Σ^ur=y} only inside Ω∖Ωr (Step 1), a set of probability zero, so P(D∩{Σ^ur=y})=P(D∩{Σur=y}) by monotonicity and additivity of P. Property (i) of a solution: u↦νuD(y) is a finite sum of the maps u↦μuD(x), each bounded and B[s,T]-measurable by property (i) for μD, hence bounded and measurable. Property (ii): let F:GN→R and write Lur for the operator of Uniqueness for the Forward Equation of a Bounded Jump-Rate Family on a Finite Set for the rates qr on GN and LuE for the one for the rates qE on E. Grouping the points of E according to Σ(x) gives νtD(F)=∑x∈EμtD(x)F(Σ(x))=μtD(F∘Σ) for every t∈[s,T], and likewise νuD(LurF)=μuD((LurF)∘Σ). By (4), for every x∈E,
(LurF)(Σ(x))=e∈A∑ge(u,x)((F∘Σ)(ϕe(x))−(F∘Σ)(x))=x′=x∑ e:ϕe(x)=x′∑ge(u,x)((F∘Σ)(x′)−(F∘Σ)(x))=LuE(F∘Σ)(x),
the middle step regrouping the labels e by the value ϕe(x) and discarding those with ϕe(x)=x, whose terms vanish. Hence νuD(LurF)=μuD(LuE(F∘Σ)), and property (ii) for μD with the function F∘Σ yields, for every t∈[s,T],
νtD(F)=μtD(F∘Σ)=μsD(F∘Σ)+∫[s,t]μuD(LuE(F∘Σ))du=νsD(F)+∫[s,t]νuD(LurF)du.
Thus (νuD)u∈[s,T] is a solution of the forward equation on [s,T] for the rates qr with rate bound l(l−1)NB, which completes the proof of claim 2.