Motivation
A nonlinear model can have many more parameters than observed predictions and still be difficult to train. Parameter count describes the size of the parameter vector, while successful fitting requires the model to move its observed outputs in the directions demanded by the residual. The Jacobian measures precisely that local ability. Its row Gram matrix is positive definite when every nonzero residual direction produces a nonzero parameter gradient.
Positive definiteness at initialization is only a beginning. If the Jacobian changes too much during training, the useful directions can disappear. A complete argument therefore needs two linked estimates: a neighborhood on which the Jacobian remains effective, and a path-length bound showing that training never leaves that neighborhood. Proving one estimate while merely assuming the other along the eventual trajectory would leave a circular argument.
We give a deterministic sufficient theorem for a twice continuously differentiable model, followed by a discrete gradient-descent version. The theorem applies when its measurable numerical hypotheses can be verified; it does not infer those hypotheses from width alone. In particular it is not a proof of every random-initialization neural tangent kernel theorem, and its classical smoothness assumptions do not cover a nonsmooth ReLU model without additional analysis.
Definitions
Definition 7.1.1
(Residual, output Jacobian, and local stability)
Let be on an open set, and fix observed labels . Use the unnormalized least-squares loss
The Jacobian has rows and columns. At an initialization , let . A positive row singular-value lower bound means for every .
This condition requires , but the converse is false. It is about the row rank and conditioning of the particular observed model, not just the dimensions. Use Euclidean norms for vectors and operator norms for matrices. Assume the closed ball around lies in , and the Jacobian is -Lipschitz on this ball. We permit for a locally linear model.
Gradient flow solves . Discrete descent uses . Because the loss is a sum rather than an average, no hidden factor appears in these equations. Averaging the loss rescales time and the permitted step accordingly.
Concept lensAnalytic
Training seen in output space
Along gradient flow the residual obeys , a linear system in output space whose matrix changes with . For a linear model the matrix is constant and : the component along an eigenvector of with eigenvalue decays like , and a zero eigenvalue leaves its component untouched. The nonlinear theorem keeps only a uniform floor on the ball, which is why its rate is .
Theorem/Proposition
Theorem 7.1.2
(Stable Jacobians keep gradient flow inside the fitting region)
Suppose , , and . Then gradient flow from exists for every , remains inside , and satisfies
It converges to a parameter with .
Theorem 7.1.3
(A discrete guarantee with an explicit containment margin)
In addition to , suppose , on , and is -Lipschitz there with . For , all gradient-descent iterates remain inside , converge to an interpolating parameter, and obey
One available smoothness bound is , or any larger positive bound.
If , both procedures are stationary at an exact fit and the conclusions are immediate. Otherwise the strict radius margins supply room for the containment arguments. The theorems guarantee a zero training loss and finite travel distance, not a minimum parameter norm, a unique parameter, or a population risk bound.
Proof sketch or proof idea
Both proofs begin from the same uniform Jacobian estimate on the ball and then add a containment argument; the discrete one must also check each step before using smoothness.
Proof
(Decay and travel up to a first exit, then rule out the exit)
For and any output vector , the reverse triangle inequality gives
Thus throughout the ball. Write and . While the flow stays in the ball, differentiating the residual gives , hence
Integration gives the exponential residual bound. Decay alone does not bound parameter motion. Put . Wherever ,
The inequality uses , and integrates to a path length at most . If the residual reaches zero, the vector field vanishes and the solution remains there, so the length estimate continues to hold.
Now remove the provisional assumption that the flow remains in the ball. The hypothesis makes the vector field locally Lipschitz, giving a unique local solution. Up to any first exit, the length bound places it in the smaller closed ball of radius , so it cannot reach the boundary of . Nor can it cease to exist at a finite time while staying in that compact inner ball: the vector field is bounded and locally Lipschitz on a neighborhood, so the finite-time limit extends the solution. Hence the trajectory is global. Finite total length makes it Cauchy as ; completeness gives a parameter limit, and continuity plus residual decay gives exact fitting.
Proof X-ray
Why the containment proof must run in this order
The hypothesis is used once, to transfer the initial row bound to every point of the declared ball at half strength. Everything later works only inside that ball, so the proof cannot start by assuming the trajectory stays there. The key move is the path-length inequality : residual decay is converted into a bound on parameter travel, because the same Gram lower bound controls both against and the speed of the flow.
The bootstrap then closes the circle. Decay and travel are proved only up to a hypothetical first exit; the strict margin places the trajectory strictly inside, so the exit never happens. Without the margin the argument stalls at the boundary, and without a controlled neighborhood an initial full row rank supplies nothing: the square-output example below has at the start and still fails to fit.
The discrete proof follows the same plan, but containment must be checked before each step, since a descent inequality on the ball says nothing about a step that might leave it.
Proof
(Check each step lies in the ball before using smoothness)
Let and . We prove by induction both containment and the partial-length estimate . If some , the iteration is stationary from then on and nothing remains to prove. The accumulated length and give
Indeed , since , and . The entire segment from to is in the convex ball, so the local smoothness bound is legitimate. It gives , and in particular . Rearranging,
This closes the partial-length induction. Also , so the descent estimate yields the geometric loss factor . The factor is nonnegative because . Summing the length estimate proves convergence of the parameters, while geometric loss decay proves interpolation at their limit.
To verify the stated , first note by integrating the Jacobian along a segment from the initialization. For two points in the ball, expand the difference of into a Jacobian difference times one residual and a fixed Jacobian times a residual difference. The two terms are bounded by and times the parameter distance. This establishes the claimed Lipschitz gradient bound without assuming the loss is globally smooth.
An alternative Gram perturbation test
Sometimes a model analysis directly estimates the observed Gram matrix. Write , with and . Expanding the product gives , so its operator norm is at most . For a unit output vector, the change in its Gram quadratic form is no larger than this number. Thus if the initial smallest Gram eigenvalue exceeds this perturbation budget, a positive lower bound persists. This is a valid alternative sufficient test, although it can be more conservative than working directly with singular values.
The test also clarifies what a large parameter space does not supply. Adding zero columns changes neither the row Gram matrix nor its smallest eigenvalue. Duplicating columns can alter scaling without adding an independent output direction. The useful property is a lower bound that controls every residual direction in the observed output space. Once that lower bound is established throughout the neighborhood, the flow proof can use it directly, with its square-root playing the role of . Establishing the neighborhood and keeping the trajectory inside it remain necessary steps.
The positivity used here lives in the output space; it does not require positive curvature in every parameter direction. When , the matrix always has null directions, yet the row Gram matrix can still be positive definite. What must be checked is the row singular values; padding the rectangular matrix with zero singular values and reading off the smallest one in parameter space would misread the theorem. Output dimension and parameter dimension play different roles in the proof.
This also explains why the theorem does not give a unique parameter. At an exact fit the residual term of the loss Hessian vanishes and remains, which may still have many flat parameter directions. The path estimates guarantee only that some nearby solution is reached; they create no identification condition for these invisible directions. Treating training success as parameter identification is a separate question.
Relation to neural tangent kernels
The Gram matrix in this proof is the finite observed Jacobian Gram matrix. For a parameterized predictor, its entries are inner products of parameter gradients at two observed inputs. This connects the argument to neural tangent kernel analysis. However, proving a positive initial eigenvalue and a small Jacobian change for a particular randomly initialized architecture requires further architecture-specific estimates.
Jacot, Gabriel and Hongler analyze an infinite-width kernel limit; their training-limit theorem states smoothness assumptions on the activation. Oymak and Soltanolkotabi give nonlinear least-squares results and distinguish smooth from nonsmooth arguments. Our elementary deterministic proof is a sufficient smooth-model statement. It supplies no numerical ReLU width threshold or probability of a successful random initialization.
Worked examples
Worked example 7.1.4
(A genuinely nonlinear, verifiable fitting neighborhood)
Take one observation, two parameters, , , and . Then , , and a valid initial lower bound is . Use , and . We have and . The smoothness bound is about , so is safe. The first step is , giving output about and residual about . The theorem guarantees convergence despite the nonlinear sine dependence; it does not say the fitted parameter is unique.
Worked example 7.1.5
(Many parameters, one missing output direction)
Let and labels . Increasing adds parameters but never separates the two output coordinates. The Jacobian has identical rows, so its Gram matrix is singular. Writing the common output as , the loss is , with minimum one. Parameter count alone cannot create the missing residual direction, and exact fitting is impossible.
Worked example 7.1.6
(Initial row rank can disappear during training)
Take , with any number of unused extra parameters, , and initial . Initially the one-row Jacobian is nonzero and . Yet no parameter fits a negative label. Gradient flow satisfies , tends to zero and has residual tending to one; its Gram value tends to zero. For and , stability requires , while the flow travel condition would require . The sufficient assumptions correctly fail. Initial positive definiteness by itself does not establish persistent positive definiteness.
Common mistakes
Counterexample mode
Capacity does not prevent a zero-gradient initialization
Let and . There are many exact fits, but at the Jacobian and gradient both vanish. Gradient flow and gradient descent remain at loss , no matter how large is. The repair is to verify a useful initialization and stable output directions, rather than infer them from representational capacity.
Common mistake
(Using a descent inequality before checking the step stays in the ball)
The smoothness constant is valid only on . Applying to a step whose endpoint may lie outside the ball is unjustified, because outside the ball the Jacobian may change faster than allows. The discrete proof therefore shows first and only then uses the inequality; this is why it needs the larger margin .
Common mistake
(Reading finite travel as minimum norm)
The theorems bound how far training moves, not which interpolating parameter it reaches. For the linear model with and , the gradient keeps the flow on the line , which meets at . The minimum-norm fit is . The minimum-norm property of linear least squares needs zero initialization (or an initialization in the row space); it is not a consequence of finite travel.
Finally, fitting arbitrary labels can include fitting arbitrary noise. Population accuracy requires a separate statistical analysis even when every optimization hypothesis is satisfied.
Summary
The Jacobian converts residuals into parameter gradients. A stable row singular-value bound makes residuals decay, and a travel estimate ensures the geometry remains valid. Continuous and discrete proofs differ in their containment margins. Width, smoothness, initialization and data geometry must be analyzed separately before applying these sufficient theorems to a neural architecture.
Exercises
Quick check
(Show that is equivalent to for every output vector .)
Solution · Answer
For every , . The matrix inequality for all is therefore the squared norm inequality, and taking square roots of these nonnegative quantities gives the equivalence.
Quick check
(Take , , , . Check both radius conditions and give the flow residual rate and the discrete total-length bound.)
Solution · Answer
We have , and . Since , the flow has residual at most . Discrete total length is at most once the additional smoothness and step conditions are supplied; radius conditions alone do not determine a safe step.
Quick check
(If the loss is divided by , how do the gradient-flow time scale and a safe discrete step change?)
Solution · Answer
Dividing the loss divides its gradient by . The new flow at time equals the original flow at time , so decay per unit time is slower by that factor. A discrete step for the averaged loss makes the same update as step for the sum loss, and the gradient Lipschitz constant is also divided by , so the safe range becomes .
Quick check
(For the square-output example with , explain why adding any number of unused parameters changes neither interpolation feasibility nor the failure of the stability conditions.)
Solution · Answer
The attainable outputs remain , so is still impossible. The Jacobian merely acquires zero columns, leaving its initial singular value and its Lipschitz constant unchanged. Consequently the incompatible requirements and persist: more coordinates have changed neither the model's image nor its relevant local geometry.