What is Isotonic Regression?

Short Answer

• Isotonic regression is similar to linear regression, except for some monotonic constraints. Additionally, linear regression is global, while isotonic regression is pairwise.

Long Answer

• Assume we have dataset \(\{(x_{1}, y_{1}), (x_{2}, y_{2}), \ldots, (x_{n}, y_{n})\}\) such that \(x_{1}<x_{2}< \cdots <x_{n}\). Isotonic regression looks for \(\beta_{1}, \beta_{2}, \ldots, \beta_{n} \in \mathbb{R}\), such that the \(\beta_{i}\)'s approximate the \(y_{i}\)'s well while being monotomically non-decreasing.

• In mathematics, the \(\beta_{i}\)'s are the solution to the optimization problem:

\[\operatorname{minimize}_ {\beta_{1}, \ldots, \beta_{n}}  \sum_{i=1}^{n}\left(y_{i}-\beta_{i}\right)^{2} \]

subject to the constraint \(\beta_{1} \leq \cdots \leq \beta_{n}\).

• Generally, we use isotonic regression when we have some preknowledge to ensure that there is a non-decreasing relationship between the predictor and response.

• Python codes for isotonic regression:
  from sklearn.isotonic import IsotonicRegression
iso_reg = IsotonicRegression().fit(X,y)
iso_reg.predict([x1, x2])