What is LARS?

Short Answer

  • LARS (least angle regression) is an algorithm for computing linear regression with \(l_{1}\) regularization stepwisely.

  • The basic steps of the LARS algorithm are:

    • Start with all coefficients \( \beta \) equal to zero.
    • Find the feature \(x_j\) most correlated with the response \(y\).
    • Increase the coefficient \(\beta_j\) in the direction of the sign of its correlation with \(y\). Calculate residuals \( r = y - \hat{y}\) along the way. Stop when some other feature \(x_k\) has as much correlation with \( r\) as \(x_j\) has.
    • Increase \((\beta_j, \beta_k )\) in their joint least squares direction, until some other feature \(x_m\) has as much correlation with the residual \(r\).
    • Increase \((\beta_j, \beta_k, \beta_m )\) in their joint least squares direction, until some other feature \(x_n\) has as much correlation with the residual \(r\).
    • Continue until: all features are in the model.
  • The figure below shows how LARS works when there are only two features.

  • Python code for LARS:
    from sklearn import linear_model
    lars = linear_model.Lars(n_nonzero_coefs = 10, normalize = True)
    lars.fit(X,y)
    lars.predict(X_test)