Gradient descent optimization methods: overview

In the world of machine learning, gradient descent is a pivotal optimization algorithm, a mathematical compass that guides you through the multidimensional terrains of data. It is a fundamental technique employed to fine-tune models, enhancing their ability to predict and infer with ever-increasing accuracy. In other words, is a mechanism by which learning is achieved.

In this topic you will become familiar with this algorithm, its possible drawbacks and the most common methods to deal with them, so let's not waste any more time with introductions!

Losses and triumphs

In machine learning you have a set of data where each observation has different features and a label, which can be a category or a number. When new observations appear, the goal of your model is to predict their label based on their features. But how to quantify how well the model assigns labels?

The loss function stands at the core of model optimization, serving as a quantifiable measure of performance. It is a mathematical way to capture the discrepancy between the predicted outputs of your model and the actual target values you aim to predict. In essence, the loss function scores the accuracy of your model, assigning a lower score to better-performing models.

A machine learning model has several parameters and once you assign them values the model is able to make a prediction. After that you can calculate the loss associated with this particular set of parameter values. So, the loss function $L$ is real-valued and each parameter is a variable. When you change the parameter values, you get a different loss. Consequently, the lower the loss, the better the parameter values. The learning processes consists in finding these optimal values.

However, there are lots of different possible values for the parameters and the loss function can be very complicated or high-dimensional, especially when there are several parameters. How to find the minimum of the loss function systematically regardless of its shape or the number of parameters? This problem is known as optimization and here is exactly where gradient descent comes into play.

How to reach the minimum

Gradient descent is a general technique that helps you out to find the minimum of $L$. When $L$ is differentiable, the derivative becomes a really useful tool for optimization since it completely describes how the function is changing in any point.

The gradient $\nabla L$ at a point $\mathbf{a}$ is a vector that points in the direction in which the $L$ changes faster, the steepest ascent in its graph. Hence, to minimize the loss, moving in the opposite direction of the gradient seems like a good idea.

To illustrate the use of the gradient, consider the loss function$L(x,y)=8 x^2+2 y^2$Since is values are non-negative its minimum is $0$ and it attains it in point $(0,0)$.

A simple calculations reveals that the gradient is given by

$\nabla L(x,y)=(16 x, 4 y) = 4 (4x, y)$Following the direction of the gradient, you go to where the function changes the most. But going in the opposite direction, that is $-\nabla L(x,y) = - 4 (4x, y)$, you get closer to the origin, which is where this function reaches its minimum:

And the same phenomenon happens for any function! Head on to the next section to find out how the gradient is used as the key piece of the gradient descent algorithm.

The gradient descent algorithm

Gradient descent is an iterative process. You start by assigning any value $\mathbf{a}$ to the parameters and then the step is:

1. Predict the target value associated with these parameter values

2. Use the prediction to compute the loss.

3. After that, calculate the gradient $\nabla L (\mathbf{a})$.

4. Move the parameters a little in the opposite direction of the gradient:$\mathbf{b} = \mathbf{a} - r \ ∇L(\mathbf{a})$

Here $\mathbf{b}$ contains the updated parameter values. You have to perform several steps until the loss function stops decreasing. Each step involves a slight adjustment of the model's parameters against the gradient, proportionally to the factor $r$ known as the learning rate. With each iteration, the model's predictions become more aligned with the desired outcomes, as the loss function's value always decreases.

Common problems

Despite its conceptual simplicity, gradient descent faces several challenges in practice:

• If your model is very complex, calculating the loss at each step could be very computationally expensive. Since you usually need thousands of steps, the training process might take a long time.

• The loss function's landscape may contain local minima —points where the function value is lower than that of the surrounding points but not the lowest overall. The algorithm might mistakenly converge to these points instead of the global minimum.

• You should take a reasonable value for the learning rate. When it's too small, the descent down the graph might take too many iterations. If the learning rate is too large, the updates can be chaotic and you might end up moving without direction.

• Other issues include slow convergence on flat regions in the loss function, known as plateaus.

In practice you must take all these factors into account, so let's see how to deal with them.

Optimization methods

To address these challenges, several variations of the gradient descent algorithm have been developed:

• Stochastic Gradient Descent (SGD): This variation considers only a single sample or a small subset of the dataset to compute the gradient and update the model parameters. As a result, computing the loss becomes faster. When the you take a single sample the method is called true SGD, otherwise it is known as mini-batch SGD. Stochastic means random, and refers to the fact that at each step you take a random sample.

• Momentum: This technique incorporates the concept of inertia from physics to help accelerate convergence, particularly in the presence of plateaus or local minima. It does so by combining the gradient of the current step with the previous step's update vector.

• Adagrad: An adaptive learning rate method, modifies the learning rate for each parameter individually, based on historical gradients. This allows for a more nuanced step size, larger for infrequent features and smaller for frequent ones, leading to more tailored updates.

There are other variants such as RMSprop and all of them are called optimization methods or optimizers. But for now it is more than enough, head on into some exercises to prove your understanding.

Conclusion

• A loss function $L$ is mathematical way to quantify the discrepancy between the predicted outputs of the model and the actual target values.

• The task of gradient descent is to minimize the loss function.

• Gradient descent is an iterative process where you approach the minimum of the function by taking small steps along the path indicated by the gradient.

• Some problems that may arise during training are the presence of multiple local minima, a very large or very small learning rate, or a very complex model.

• There are various methods to deal with algorithm problems, such as SGD, Momentum or Adagrad.