Linear Regression Intuition: Scaling and Gradient Descent

Why I wrote 50 lines of code when import sklearn takes three. I’ve been building Linear Regression from scratch to really get an intuition of how these algorithms work. Here is the realization that hit me: The Learning Rate is not absolute; it is relative to the scale of your data. Because I was using raw salary data ($100k+), the gradients were massive. Multiplying a massive gradient by 0.01 still resulted in a huge step size, causing the algorithm to overshoot the minima entirely. This is exactly why libraries like Scikit-Learn emphasize preprocessing pipelines. Without normalizing or standardizing your features, Gradient Descent is fighting the scale of your own data. Abstraction is great for productivity, but implementation is essential for intuition. #MachineLearning #Algorithms #DataScience #ArtificialIntelligence #Python #ScikitLearn

  • No alternative text description for this image

Does the circles represents confidence intervals/credible regions, in any sense? I often come across similar looking contour plots generated for MCMC and that represents Credible regions.

Like
Reply

Interesting I would appreciate you to tell me how you started learning the algorithms from scratch, I mean there are so many resources and I want to learn but I get really confused

Like
Reply

On top of that you will know what to tweak when your model is not performing well. Once learned how the algorithm works behind the scenes, the wisdom that you gain, will guide you through further.

Like
Reply

Interesting 🤔 I mean if you do not normalise it, the step would be big Coz step = -learning_rate times gradient But shouldn't that balance out given the path would be non normalized 🤔

Like
Reply

Abstractions are not always helpful especially when it comes to understanding low-level details!

Like
Reply

To get better understanding of regression, learn the Gaussian process and Cholesky decomposition.

See more comments

To view or add a comment, sign in

Explore content categories