mathematics//optimization//nonlinear least squares//Levenberg-Marquardt algorithm

The Levenberg-Marquardt algorithm is an iterative solver for nonlinear least squares that blends the Gauss-Newton step with a gradient-descent step through one damping parameter, adjusted at every iteration, and it is the default engine of curve fitting and calibration: SciPy's `curve_fit` uses it for unconstrained problems, Ceres uses it by default (and g2o offers it) to solve SLAM graphs and bundle adjustment, and it fits camera intrinsics, motor models and battery curves to test data. It takes a model, its residuals, their Jacobian and a starting guess, and returns the parameters that minimize the sum of squared residuals near that guess.

The Levenberg-Marquardt algorithm is an iterative solver for nonlinear least squares that blends the Gauss-Newton step with a gradient-descent step through one damping parameter, adjusted at every iteration, and it is the default engine of curve fitting and calibration: SciPy's curve_fit uses it for unconstrained problems, Ceres uses it by default (and g2o offers it) to solve SLAM graphs and bundle adjustment, and it fits camera intrinsics, motor models and battery curves to test data. It takes a model, its residuals, their Jacobian and a starting guess, and returns the parameters that minimize the sum of squared residuals near that guess.

Gauss-Newton is fast when the linearization is good and reckless when it is not: far from the solution, its step can overshoot into a region where the error is larger. Gradient descent is safe and slow. Levenberg-Marquardt solves, at each iteration,

(JTJ+λD)δ=−JTr,\left(J^{\mathsf T}J+\lambda D\right)\delta=-J^{\mathsf T}r,(JTJ+λD)δ=−JTr,

with rrr the residuals, δ\deltaδ the step and DDD the identity or the diagonal of JTJJ^{\mathsf T}JJTJ. With λ\lambdaλ near zero this is the Gauss-Newton step; with λ\lambdaλ large the λD\lambda DλD term dominates and the step becomes a short move down the gradient. The rule is simple: if the step lowered the error, accept it and shrink λ\lambdaλ (trust the model more); if it raised the error, reject it and grow λ\lambdaλ (take a smaller, safer step).

The damping is a trust region in disguise.

A large λ\lambdaλ keeps the step short, inside the zone where the linearized model can be believed, and lets it lengthen as the model proves itself; near the solution the algorithm runs at Gauss-Newton speed, far from it at gradient-descent safety, with no switch to decide by hand.

The damping also rescues a badly conditioned problem. Adding λD\lambda DλD makes JTJJ^{\mathsf T}JJTJ invertible even when two parameters are nearly indistinguishable, the same move as ridge regression; it hides the symptom without curing it, and the cure for parameters the data cannot separate is a better experiment (condition number).

It finds a local minimum. A cooling curve fitted from a starting time constant a hundred times too large can converge somewhere absurd, so a rough physical guess (read off a plot, or from a linear fit of a transformed model) is part of using it; with outliers in the data, a robust loss or a first pass of RANSAC comes first.

It is Newton's method specialized to sums of squares, needing only first derivatives; for problems that are not sums of squares, or that have bounds and constraints, other trust-region or interior-point solvers take over (SciPy's least_squares switches to a trust-region reflective method when bounds are given).