Follow

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use
Contact

Linear regression with gradient descent won't converge

I’ve written linear regression from scratch. The fit function calculates partial derivatives at each epoch for slope (m) and bias (b) and updates these variables using their partial derivatives.

My loss decreases, but it gets stuck at a curious place.

The code:

MEDevel.com: Open-source for Healthcare and Education

Collecting and validating open-source software for healthcare, education, enterprise, development, medical imaging, medical records, and digital pathology.

Visit Medevel

class LinearRegression:

    def __init__(self):
        self.m = 3
        self.b = 2

    def fit(self, x, y, epochs, lr):

        for epoch in range(epochs):
            pred = (self.m * x) + self.b

            cur_loss = np.sum((y - pred)**2)
            if epoch % 1000 == 0:
                print(cur_loss)

            pd_wrt_m = (-2/x.size) * np.sum((y - pred) * x) 
            pd_wrt_b = (-2/x.size) * np.sum(y - pred)

            self.m -= (lr * pd_wrt_m)
            self.b -= (lr * pd_wrt_b)


    def predict(self, x):
        return (self.m * x) + self.b

lr_model = LinearRegression()
lr_model.fit(x, y, 10000, 0.01)

Y_pred = lr_model.predict(x)

plt.scatter(x, y)
plt.plot([min(x), max(y)], [min(Y_pred), max(Y_pred)], color='red')  # regression line
plt.show()

And this is the output when the above code is executed.

output

I’ve tried smaller and larger learning rates, smaller and larger epoch counts, but it seems to be the case that wherever the initial m and b values start out, they always stop converging at this particular position.

I’ve reread my code and could not find a mistake. What could I be doing wrong?

>Solution :

You wrote max(y) instead of max(x) lol.
Replace the plt.plot line with:

plt.plot([min(x), max(x)], [min(Y_pred), max(Y_pred)], color='red')  # regression line
Add a comment

Leave a Reply

Keep Up to Date with the Most Important News

By pressing the Subscribe button, you confirm that you have read and are agreeing to our Privacy Policy and Terms of Use

Discover more from Dev solutions

Subscribe now to keep reading and get access to the full archive.

Continue reading