The Hessian

The best and most comprehensive description of the space is given by the quadratic approximation of the topology which is generated from the combination of the function value, the gradient, and the Hessian. From the second-order Taylor series expansion the quadratic model of the space is

$$\approx {\tfrac{{1}}{{2}}}$$ (14.5)

where ∇²f_k is the Hessian, which is the symmetric matrix of second partial derivatives of the function, at the position θ_k. The Hessian is the matrix of second partial derivatives and is defined as

$$\begin{pmatrix} \frac{\partial^2}{{\partial \theta_1}^2} & \frac{... ...\theta_2} & \ldots & \frac{\partial^2}{{\partial \theta_n}^2} \\ \end{pmatrix}$$ (14.6)

The order in which the partial derivatives are calculated is inconsequential, hence the Hessian is symmetric.

As the Hessian is computationally expensive a number of optimisation algorithms try to approximate it, the BFGS algorithm being a notable example. The most powerful minimisation algorithm for model-free analysis - Newton optimisation - requires the value, gradient, and Hessian at the current parameter values.