Linear Regression Regularization bias about ml-from-scratch

[adityach007]

Your observation about the potential impact of including bias in L1 regularization for Lasso regression is correct. The bias term (typically the zeroth weight) should sometimes be treated differently to avoid inadvertently penalizing it excessively when applying L1 regularization.

Your suggested modification, excluding the bias term from the norm calculation in L1 regularization, is a sound approach to address this issue. It effectively adjusts the regularization calculation to avoid penalizing the bias term along with other weights.

class L1Regularization():

def __init__(self, alpha):
    self.alpha = alpha

def __call__(self, w):
    return self.alpha * np.linalg.norm(w[1:], ord=1)  

def grad(self, w):
    grad = np.zeros_like(w)
    grad[1:] = self.alpha * np.sign(w[1:]) 
    return grad