Abstract
Gradient Boosted Decision Trees (GBDT) is a very
successful ensemble learning algorithm widely
used across a variety of applications. Recently, several variants of GBDT training algorithms and implementations have been designed and heavily optimized in some very popular open sourced toolkits including XGBoost, LightGBM and CatBoost.
In this paper, we show that both the accuracy and
efficiency of GBDT can be further enhanced by
using more complex base learners. Specifically,
we extend gradient boosting to use piecewise linear regression trees (PL Trees), instead of piecewise constant regression trees, as base learners. We
show that PL Trees can accelerate convergence of
GBDT and improve the accuracy. We also propose
some optimization tricks to substantially reduce the
training time of PL Trees, with little sacrifice of
accuracy. Moreover, we propose several implementation techniques to speedup our algorithm on
modern computer architectures with powerful Single Instruction Multiple Data (SIMD) parallelism.
The experimental results show that GBDT with PL
Trees can provide very competitive testing accuracy with comparable or less training time