Dictionary of Applied Machine Learning

gradient

Typeset PDF version — the authoritative form of this entry

Python demo — a script that recomputes what this entry states and prints one line per check

The gradient of a real-valued function is a vector that determines the local linear approximation of the function. The entries of the gradient are the partial derivatives of the function. However, the existence of partial derivatives does not imply the existence of a gradient, unless the function is convex. Geometrically, a nonzero gradient is orthogonal to the level sets and points in the direction of steepest ascent. In machine learning (ML), gradients of the empirical risk minimization (ERM) objective drive gradient descent (GD) methods and are computed for deep networks by backpropagation.

Definition

B-partialsThe gradient of a real-valued function $f: \reals^{\featuredim} \rightarrow \reals: \weights \mapsto f(\weights)$ determines a local linear approximation of $f$. Formally, the gradient of $f$ at a point $\weights' \in \reals^{\featuredim}$ is a vector $\vg \in \reals^{\featuredim}$ such that \[ \lim_{\weights \rightarrow \weights'} \frac{f(\weights) - \big( f(\weights') + \vg^{\top} (\weights - \weights') \big)}{\normgeneric{\weights - \weights'}{2}} = 0 \text{.} \] If such a vector exists, it is unique and denoted by $\nabla f(\weights')$ or $\nabla f(\weights) \big|_{\weights'}$ (Rudin, 1976, Ch. 9). The function $f(\weights') + \big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$ is the local linear approximation of $f$ at $\weights'$ (see Fig. 1). The entries of the gradient are the partial derivatives of $f$, $\nabla f(\weights) = \big( {\partial f}/{\partial \weight_{1}}, \ldots, {\partial f}/{\partial \weight_{\featuredim}} \big)^{\top}$.

Figure 1 of the entry gradient
Figure 1: A differentiable function $f: \reals \rightarrow \reals$ and its local linear approximation at a point $\weights'$. The slope triangle shows the displacement $\weights - \weights'$ and the resulting change $\big(\nabla f(\weights')\big)^{\top} (\weights - \weights')$ of the local linear approximation
In general, the existence of all partial derivatives of $f$ at a point does not guarantee that the gradient exists there (Rudin, 1976, Ch. 9). For a convex function $f: \reals^{\featuredim} \rightarrow \reals$, however, it does: if the partial derivatives $\partial f / \partial \weight_{\featureidx}$, for $\featureidx = 1, \ldots, \featuredim$, exist at a point $\weights'$, then $f$ is differentiable at $\weights'$, and the gradient $\nabla f(\weights')$ is the vector of these partial derivatives (Rockafellar, 1970, Sect. 25).

B-hilbertThe definition carries over to a real-valued function $f: \hilbertspace \rightarrow \reals$ on a Hilbert space $\hilbertspace$: the inner product $\innerprod{\vg}{\weights - \weights'}$ of $\hilbertspace$ replaces the term $\vg^{\top} (\weights - \weights')$, and the norm of $\hilbertspace$ replaces the Euclidean norm (Bauschke and Combettes, 2011).

B-backpropThe gradient has a geometric interpretation. At every point where $f$ is differentiable and $\nabla f \neq \mathbf{0}$, the gradient is orthogonal to the level set of $f$ through that point, and it points in the direction of steepest ascent of $f$. The negative gradient $-\nabla f(\weights')$ points in the direction of steepest descent (see Fig. 2; Boyd and Vandenberghe, 2004, Sect. 9.4). At a local minimum of a differentiable function $f: \reals^{\featuredim} \rightarrow \reals$, there cannot be any direction of descent and consequently the gradient must vanish (see zero-gradient condition).

Figure 2 of the entry gradient
Figure 2: Level sets $f(\weights) = \text{const.}$ of a differentiable function $f$ of two variables, with a minimum at $\widehat{\weights}$. The gradient $\nabla f(\weights')$ (solid) is orthogonal to the level set through $\weights'$ and points in the direction of steepest ascent. The negative gradient $-\nabla f(\weights')$ (dashed) points in the direction of steepest descent, toward smaller values of $f$. Gradient descent (GD) repeatedly steps along this direction. Data generated by pythondemos/gradient.py
Gradients are instrumental for the training of machine learning (ML) models. They guide the update of model parameters in order to minimize the incurred loss on a training set $\trainset = \big\{ \big(\featurevec^{(\sampleidx)}, \truelabel^{(\sampleidx)}\big) \big\}_{\sampleidx=1}^{\samplesize}$. As a case in point, linear regression minimizes the objective function \[ f(\weights) \defeq \frac{1}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \big( \truelabel^{(\sampleidx)} - \weights^{\top} \featurevec^{(\sampleidx)} \big)^{2} \text{.} \] This function is convex and differentiable, with gradient \[ \nabla f(\weights) = - \frac{2}{\samplesize} \sum_{\sampleidx=1}^{\samplesize} \big( \truelabel^{(\sampleidx)} - \weights^{\top} \featurevec^{(\sampleidx)} \big) \featurevec^{(\sampleidx)} \text{.} \] For a deep artificial neural network (ANN) with differentiable activation functions, the empirical risk minimization (ERM) objective function is also differentiable. The gradient of this objective function can be computed by backpropagation (Goodfellow et al., 2016, Sect. 6.5; Rumelhart et al., 1986).

See also: function, vector, differentiable, partial derivative, gradient descent, zero-gradient condition, convex, Hilbert space.

References

  1. Rudin (1976). Principles of Mathematical Analysis. mcgraw.
  2. Rockafellar (1970). Convex Analysis. Princeton Univ. Press. doi.org/10.1515/9781400873173
  3. Bauschke and Combettes (2011). Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer Science+Business Media. doi.org/10.1007/978-1-4419-9467-7
  4. Boyd and Vandenberghe (2004). Convex Optimization. Cambridge Univ. Press. doi.org/10.1017/CBO9780511804441
  5. Goodfellow et al. (2016). Deep Learning. MIT Press.
  6. Rumelhart et al. (1986). Learning Representations by Back-Propagating Errors. Nature. doi.org/10.1038/323533a0

Cite this entry

@misc{dictml_gradient,
  author = {Jung, Alexander},
  title = {gradient},
  howpublished = {Dictionary of Applied Machine Learning (course edition)},
  year = {2026},
  doi = {10.5281/zenodo.21569296},
  note = {ISBN 978-952-64-3013-3, CC BY 4.0, retrieved 2026-08-21},
  url = {https://dictionaryofml.org/terms/gradient.html}
}