mathematics//calculus//Hessian
The Hessian is the square matrix of second-order partial derivatives of a scalar function, and it captures the curvature of the function at a given point. If a function has \(n\) parameters, its Hessian is an \(n\times n\) matrix whose entry \((i,j)\) is the second derivative with respect to parameters \(i\) and \(j\), \(H_{ij}=\partial^2 f/\partial\theta_i\,\partial\theta_j\).
The Hessian is the square matrix of second-order partial derivatives of a scalar function, and it captures the curvature of the function at a given point. If a function has nnn parameters, its Hessian is an n×nn\times nn×n matrix whose entry (i,j)(i,j)(i,j) is the second derivative with respect to parameters iii and jjj, Hij=∂2f/∂θi ∂θjH_{ij}=\partial^2 f/\partial\theta_i,\partial\theta_jHij=∂2f/∂θi∂θj.
In deep learning, the Hessian of the loss function describes the local curvature of the loss landscape. Its eigenvalues reveal whether a critical point is a minimum (all positive), a maximum (all negative), or a saddle point (mixed).
H = [[a, b], [b, c]][[2.0, 0.6], [0.6, 1.0]] λ12.28 λ20.72 f(x, y) = ½ (a x² + 2b xy + c y²) has this Hessian at every point. Both eigenvalues positive: the surface curves up in every direction. The origin is a minimum.
Turn the surface with a finger, then pull ccc below zero: one eigenvalue changes sign and the bowl opens into a saddle; at exactly zero it becomes a flat trough the second derivatives cannot judge.
The same eigenvalues say how hard the function is to minimize. Near a minimum every smooth loss looks like the quadratic 12θ⊤Hθ\tfrac12\theta^\top H\theta21θ⊤Hθ, and gradient descent on it multiplies each eigen-direction by 1−ηλi1-\eta\lambda_i1−ηλi per step: it converges only if the learning rate satisfies η<2/λmax\eta<2/\lambda_{\max}η<2/λmax, and progress along the flattest direction, λmin\lambda_{\min}λmin, sets how long it takes. A large ratio κ=λmax/λmin\kappa=\lambda_{\max}/\lambda_{\min}κ=λmax/λmin (the condition number of the Hessian) is a long narrow bathtub that the optimizer bounces across while crawling along its floor, which is why standardizing features, momentum and Adam help, and why Newton's method, which divides the gradient by the Hessian, needs no such tuning on a quadratic. The Hessian is the Jacobian of the gradient, one level of derivative higher.
The Hessian of a neural network is impractically large (for a model with 175 billion parameters, the Hessian would be a matrix of 175 billion by 175 billion entries). In practice, only the extreme eigenvalues are computed (using iterative methods like Lanczos), which is enough to characterize the local geometry.