ML//supervised learning//decision tree//gradient boosting

Gradient boosting is an ensemble method that builds many shallow decision trees in sequence, each correcting the residual errors of the ones before it, and it is the industry default for tabular data: quality rejects from forty process parameters, failure risk from usage and temperatures, demand from calendars and weather. Each new tree, typically three to eight levels deep, is trained on what the current ensemble still gets wrong, and its prediction is added with a small weight:

Gradient boosting is an ensemble method that builds many shallow decision trees in sequence, each correcting the residual errors of the ones before it, and it is the industry default for tabular data: quality rejects from forty process parameters, failure risk from usage and temperatures, demand from calendars and weather. Each new tree, typically three to eight levels deep, is trained on what the current ensemble still gets wrong, and its prediction is added with a small weight:

Fm(x)=Fm−1(x)+ν hm(x),hm(xi)≈−∂ ℓ(yi,F)∂F∣F=Fm−1(xi).F_m(x)=F_{m-1}(x)+\nu\,h_m(x),\qquad h_m(x_i)\approx-\frac{\partial\,\ell\bigl(y_i,F\bigr)}{\partial F}\Big|_{F=F_{m-1}(x_i)}.Fm​(x)=Fm−1​(x)+νhm​(x),hm​(xi​)≈−∂F∂ℓ(yi​,F)​​F=Fm−1​(xi​)​.

Fm−1F_{m-1}Fm−1​ is the model with m−1m-1m−1 trees, hmh_mhm​ the new tree, fitted to the negative gradient of the loss at each example (with squared loss, simply the residual yi−Fm−1(xi)y_i-F_{m-1}(x_i)yi​−Fm−1​(xi​)), and ν\nuν, the learning rate, is typically 0.01 to 0.1. It is gradient descent in the space of functions, one tree per step: a committee of modest members, each fixing only what the previous ones got wrong, that ends up surprisingly competent.

On a table, boosting with good features is the rival to beat.

Thresholds need no scaling, irrelevant columns are never chosen, interactions are captured without being written and missing values are routed down a branch. Published comparisons keep finding it matching or beating deep networks on medium-sized tables at a fraction of the compute. Trees do not extrapolate, and that limit travels with it.

It is cheap to train and to run. XGBoost, LightGBM and CatBoost fit millions of rows in minutes on a CPU. A model of 500 trees of depth 6 is about 3,000 comparisons per prediction, tens of microseconds on an industrial PC, and it can be translated into plain code for a PLC.

It is often the real rival to an LSTM on small or medium tabular time-series problems, given lag features, and frequently wins there.

It inherits the blind spots of trees. Outside the training range it returns the last step (a bearing never seen above 70 °C is treated at 90 °C as at 70 °C), and its piecewise-flat output gives an optimizer no useful gradient. Feeding it physics-based features shrinks both problems, because a feature that already discounts the operating point keeps the inputs inside a range the trees have seen.

Physics can be imposed. Monotonic constraints force the prediction to move in one direction with a feature (failure risk never falls as usage hours rise), and SHAP values split each prediction among the features to explain it to the engineer who has to act on it.

It is prone to overfitting without regularization (shallow trees, a small ν\nuν, subsampling) and early stopping on a validation fold, which also chooses the number of trees.

The sibling that averages instead of chaining is the random forest, more robust without tuning and usually a little less accurate.