Proof of The Score Integrates by Parts Against Every Compactly Supported Continuously Differentiable Vector Field

Each result cited is universally quantified over the data in its own statement. Write $\xi=\xi_{\mu}$, $T=T_{\mu}$ for the tangent space (a closed linear subspace by Basic Properties of the Tangent Space: Closed Subspace, the Identity Map Belongs to It, Second-Moment Limits, and Representation of Bounded Functionals on Gradients §closed), $P=P_{T}$ for the orthogonal projection onto it (Real Hilbert Spaces: Standing Notation and Background §projections), and $\lVert\cdot\rVert=\lVert\cdot\rVert_{\mu}$.

Step 0 (the fields). Let $\mathcal{C}$ be the set of maps $\zeta:\mathbb{R}^{d}\to\mathbb{R}^{d}$ whose components are of class $C^{1}$ and compactly supported. For $\zeta\in\mathcal{C}$, each component $\zeta_{i}$ is continuous (Euclidean Space is Open in Itself, and $C^k$ Maps are Continuous, claim 3) and vanishes outside a bounded set (Compact Support on $\mathbb{R}^n$ Means Vanishing Outside a Bounded Set). On a closed ball containing that set it is bounded (Extreme Value Theorem on a Compact Subset of a Metric Space and A Closed Euclidean Ball is Convex and Compact), so it is bounded everywhere. Likewise each $\partial_{i}\zeta_{i}$ is continuous and compactly supported by Integration by Parts on Euclidean Space Against a Compactly Supported Function of Class $C^{1}$ §vanishing, hence bounded. So $\zeta$ and $\operatorname{div}\zeta$ are Borel (Probability Measures on Euclidean Space and Random Vectors: Standing Notation §borel-maps) and bounded; $\zeta\in L^{2}(\mu;\mathbb{R}^{d})$ and $\operatorname{div}\zeta$ is integrable with respect to every probability measure. The set $\mathcal{C}$ is closed under linear combinations, $\operatorname{div}$ is linear on it, and $L(\zeta)=-\int\operatorname{div}\zeta\,d\mu$ is linear on $\mathcal{C}$ (claim 2 of Linearity and Monotonicity of the Lebesgue Integral). For a test function $\psi$ one has $\nabla\psi\in\mathcal{C}$, since its components are test functions (smooth by Smooth Map on a Euclidean Open Set, compactly supported by The Gradient of a Test Function is Bounded and Square-Integrable, and Its Laplacian Bounded and Integrable, Against Every Probability Measure §gradient), and $\operatorname{div}\nabla\psi=\Delta\psi$ (The Laplacian of a Twice Continuously Differentiable Function §laplacian). So $L(\nabla\psi)=\langle\xi,\nabla\psi\rangle$ by Finite Fisher Information, the Score and the Fisher Information of a Probability Measure §score.

Step 1 (smoothed identity). Let $\zeta\in\mathcal{C}$ and $0<s\le\tfrac12$, and let $\rho_{s}$, $\mu_{s}$ and $r_{s}$ be as in Gaussian Smoothing of a Measure with Finite Second Moment: Positive Density, Finite Entropy, a Tangent Score, Non-Increasing Fisher Information and Exponential Integrability. By Gaussian Smoothing of a Measure with Finite Second Moment: Positive Density, Finite Entropy, a Tangent Score, Non-Increasing Fisher Information and Exponential Integrability §score, $\xi_{\mu_{s}}=r_{s}$. By claim 3 of Image Measures, Measures with Densities, and Change of Variables and Integration by Parts on Euclidean Space Against a Compactly Supported Function of Class $C^{1}$ §parts (with $f=\rho_{s}$, of class $C^{1}$ by Gaussian Smoothing of a Measure with Finite Second Moment: Positive Density, Finite Entropy, a Tangent Score, Non-Increasing Fisher Information and Exponential Integrability §density, and $g=\zeta_{i}$),

With Gaussian Smoothing of a Measure with Finite Second Moment: Positive Density, Finite Entropy, a Tangent Score, Non-Increasing Fisher Information and Exponential Integrability §fisher and The Cauchy-Schwarz Inequality in a Real Inner Product Space in $L^{2}(\mu_{s};\mathbb{R}^{d})$ this gives

Step 2 (removing the smoothing). Let $\phi$ be continuous, vanishing outside $\bar{B}(0_{\mathbb{R}^{d}},R)$, with $|\phi|\le M_{\phi}$; this covers $\phi=\operatorname{div}\zeta$ and $\phi=\lVert\zeta\rVert^{2}$. By Heine-Cantor Theorem: Continuity on a Compact Subset Implies Uniform Continuity on the compact ball $\bar{B}(0_{\mathbb{R}^{d}},R+1)$, for every $\epsilon>0$ there is $\delta\in(0,1]$ with $|\phi(x)-\phi(x')|\le\epsilon$ whenever $\lVert x-x'\rVert\le\delta$. This holds for all such pairs: if one point is outside that ball, both are outside $\bar{B}(0_{\mathbb{R}^{d}},R)$ and $\phi$ vanishes at both. By Gaussian Smoothing of a Probability Measure on Euclidean Space: Regularity, Mass, Duality, Approximation of a Bounded Function with a Modulus of Continuity, and the Pairing Identity §duality and Gaussian Smoothing of a Probability Measure on Euclidean Space: Regularity, Mass, Duality, Approximation of a Bounded Function with a Modulus of Continuity, and the Pairing Identity §approximation (in dimension $q=d$),

The right side is at most $2\epsilon$ once $s\le\epsilon\delta^{2}(2M_{\phi}d+1)^{-1}$. So along $s_{n}=(2n)^{-1}$ one has $\int\phi\,d\mu_{s_{n}}\to\int\phi\,d\mu$ by Limit of a Sequence of Real Numbers, since $s_{n}$ falls below any positive bound for all large $n$ (The Archimedean Property of the Real Numbers). Passing to the limit in (1) along $s_{n}$ (Order Properties of Limits of Real Sequences, and continuity of the square root) gives

Step 3 (the orthogonal component). Let $w=\eta-P\eta$, which is orthogonal to $T$ (Real Hilbert Spaces: Standing Notation and Background §projections). Since $\xi\in T$, also $\langle\xi,w\rangle=0$. The gradients of test functions are dense in $T$ (Basic Properties of the Tangent Space: Closed Subspace, the Identity Map Belongs to It, Second-Moment Limits, and Representation of Bounded Functionals on Gradients §closed), so there are test functions $\psi_{n}$ and $\varphi_{n}$ with $\nabla\psi_{n}\to P\eta$ and $\nabla\varphi_{n}\to\xi$ in $L^{2}(\mu;\mathbb{R}^{d})$ (Sequential Characterization of the Closure in a Metric Space). Let $a=L(\eta)-\langle\xi,P\eta\rangle$ and let $\tau\in\mathbb{R}$. The field $\zeta_{n}=\nabla\varphi_{n}+\tau(\eta-\nabla\psi_{n})$ lies in $\mathcal{C}$, and by Step 0

As $n\to\infty$, the right side tends to $\lVert\xi\rVert^{2}+\tau a$, and $\lVert\zeta_{n}\rVert\to\lVert\xi+\tau w\rVert$, by continuity of the inner product and the norm (The Norm Metric of a Real Inner Product Space: Triangle Inequalities, Limits and Continuity §continuity). So (B) gives

The equality holds because $\langle\xi,w\rangle=0$. The last inequality holds because, for $\alpha,\beta\ge0$, $\alpha^{2}(\alpha^{2}+\beta)\le(\alpha^{2}+\tfrac12\beta)^{2}$, and squares are monotone on nonnegative numbers (Monotonicity of Squaring on the Nonnegative Elements of an Ordered Field). Thus $\tau a\le\tfrac12\tau^{2}\lVert w\rVert^{2}$ for every real $\tau$. Taking $\tau=\pm n^{-1}$ gives $|a|\le\tfrac12n^{-1}\lVert w\rVert^{2}$ for every $n\in\mathbb{N}$, so $a=0$ (Comparison of Real Numbers with Arbitrary Positive Slack §vanishing with The Archimedean Property of the Real Numbers). Therefore