cft

A Beginners Guide to the Gradient Descent Algorithm

The gradient descent algorithm is an approach to find an optimal solution for a given dataset.


user

Ashwin Sharma P

3 years ago | 2 min read

The gradient descent algorithm is an approach to find the minimum point or optimal solution for a given dataset. It follows the steepest descent approach. That is it moves in the negative gradient direction to find the local or global minima, starting from a random point. We use gradient descent to reach the lowest point of the cost function.

In Machine Learning, it is used to update the coefficients of our model. It also ensures that the predicted output value is close to the expected output.

For example, in Linear Regression, where we separate the output using a linear equation.

Let us say we have an equation y=mx+c.

Here, m stands for slope and c for the y-intercept. These two values can be optimized and the error in the cost function (the difference between expected and predicted output) can be reduced using the gradient descent algorithm.

Weight updation

So let us see how weight updation works in gradient descent. Let us consider a graph of cost function vs weight. For improving our model, bringing down the value of the cost function is essential. We consider the lowest point in the graph as the winner since the cost function would be minimal at this point.

weight updation
weight updation

In the above diagram, we can see that with each iteration, the function tries to bring down the cost value. But that is not the case in real-world datasets. In real-world cases, it moves in a zig-zag manner for most of the datasets. The graph for real-world cases is shown below.

finding the global minimum point
finding the global minimum point

What is the maths behind it?

The weight updation takes place by decrementing the cost function in steps of the gradient (derivative) of the weight function. The equation used for weight updation is:

equation
equation

Here ω corresponds to the weight and η the learning rate, which determines by what value the descent is made in each iteration.

For every cross mark shown in the graph, we calculate the slope. According to the slope value, we update the weights in the above equation.

Conclusion

Here is a sequence diagram to brief the process of updating the weights. This update in weights leads to a reduction in the cost function.

flowchart
flowchart

Hope you learnt something from this article Thank you for taking the time to read it.

Upvote


user
Created by

Ashwin Sharma P

Loves Artificial Intelligence, philosophy and reading


people
Post

Upvote

Downvote

Comment

Bookmark

Share


Related Articles