What is LARS?

Short Answer

• LARS (least angle regression) is an algorithm for computing linear regression with \(l_{1}\) regularization stepwisely.

• The basic steps of the LARS algorithm are:

  • Start with all coefficients \( \beta \) equal to zero.
  • Find the feature \(x_j\) most correlated with the response \(y\).
  • Increase the coefficient \(\beta_j\) in the direction of the sign of its correlation with \(y\). Calculate residuals \( r = y - \hat{y}\) along the way. Stop when some other feature \(x_k\) has as much correlation with \( r\) as \(x_j\) has.
  • Increase \((\beta_j, \beta_k )\) in their joint least squares direction, until some other feature \(x_m\) has as much correlation with the residual \(r\).
  • Increase \((\beta_j, \beta_k, \beta_m )\) in their joint least squares direction, until some other feature \(x_n\) has as much correlation with the residual \(r\).
  • Continue until: all features are in the model.

• The figure below shows how LARS works when there are only two features.
  

• Python code for LARS:
  from sklearn import linear_model
lars = linear_model.Lars(n_nonzero_coefs = 10, normalize = True)
lars.fit(X,y)
lars.predict(X_test)