"LAR uses least squares directions in the active set of variables. Lasso uses least square directions; if a variable crosses zero, it is removed from the active set. Boosting uses non-negative least squares directions in the active set."
"the bias of the 1-nearest-neighbor estimate is often low, but the variance is high."
"Monte Carlo is an extremely bad method; it should be used only when all alternative methods are worse"
"Bagging; or Bootstrap AGGregatING, is an extension of bootstrapping to classification and regression problems. The main idea is to sample with replacement from the training data so that we now have B training data sets, each having n′≤n observations. The machine-learning algorithm is trained on each"
"Both k-nearest neighbors and least squares end up approximating conditional expectatios by averages."
"However, with a 0 − 1 outcome, this computation simplifies. We order the predictor classes according to the proportion falling in outcome class 1. Then we split this predictor as if it were an ordered predictor."