Regression

Regression refers to the supervised learning task of predicting a continuous target value $y \in \mathbb{R}$ from a set of input features $\boldsymbol{x}$, as opposed to predicting a discrete class label (classification). Given training data $\{(\boldsymbol{x}_i, y_i)\}_{i=1}^N$, a regression model learns a function $\hat{y} = f(\boldsymbol{x})$ that generalizes well to new, unseen inputs.

Performance is typically measured by the discrepancy between predictions $\hat{y}_i$ and true values $y_i$, most commonly the mean squared error,

or the coefficient of determination, $R^2$, which expresses the fraction of the variance in the target explained by the model (1.0 = perfect prediction, 0.0 = no better than predicting the mean).

Many of the methods in this course can be used for regression as well as classification (neural networks, decision trees, random forests, gradient boosting, SVMs), simply by changing the loss function and the output layer/leaf prediction from a class label to a real number. Gaussian process regression, covered next, is intrinsically a regression method, and additionally provides a principled estimate of predictive uncertainty alongside each prediction.