– 开发了多目标树模型,可以为多目标回归、多标签分类等建立一个树,可以防止过拟合、产生更小的模型,并考虑目标之间的相关性。
– 添加了新的device参数,用于选择运行设备,取代之前的gpu相关参数。
– hist成为默认的树构建算法,之前会根据数据和环境选择approx或exact,现在默认hist可以提高效率。
– 支持在GPU上使用approx树构建算法。性能还未充分优化,但已具备大部分功能。
– 为hist新增最大缓存节点数参数,控制CPU缓存占用,防止过度缓存占用过多内存。
– 优化了外部内存支持,使用内存映射替代文件IO,大幅提升hist性能,减少CPU内存占用。
– 新增学习排序任务实现,支持不同的组对构建策略、每组样本数控制、无偏排序学习等。
2.0.0 (2023 Sep 12)
We are excited to announce the release of XGBoost 2.0. This note will begin by covering some overall changes and then highlight specific updates to the package.
Initial work on multi-target trees with vector-leaf outputs
We have been working on vector-leaf tree models for multi-target regression, multi-label classification, and multi-class classification in version 2.0. Previously, XGBoost would build a separate model for each target. However, with this new feature that’s still being developed, XGBoost can build one tree for all targets. The feature has multiple benefits and trade-offs compared to the existing approach. It can help prevent overfitting, produce smaller models, and build trees that consider the correlation between targets. In addition, users can combine vector leaf and scalar leaf trees during a training session using a callback. Please note that the feature is still a working in progress, and many parts are not yet available. See #9043 for the current status. Related PRs: (#8538, #8697, #8902, #8884, #8895, #8898, #8612, #8652, #8698, #8908, #8928, #8968, #8616, #8922, #8890, #8872, #8889, #9509) Please note that, only the hist
(default) tree method on CPU can be used for building vector leaf trees at the moment.
New device
parameter.
A new device
parameter is set to replace the existing gpu_id
, gpu_hist
, gpu_predictor
, cpu_predictor
, gpu_coord_descent
, and the PySpark specific parameter use_gpu
. Onward, users need only the device
parameter to select which device to run along with the ordinal of the device. For more information, please see our document page (https://xgboost.readthedocs.io/en/stable/parameter.html#general-parameters) . For example, with device="cuda", tree_method="hist"
, XGBoost will run the hist
tree method on GPU. (#9363, #8528, #8604, #9354, #9274, #9243, #8896, #9129, #9362, #9402, #9385, #9398, #9390, #9386, #9412, #9507, #9536). The old behavior of gpu_hist
is preserved but deprecated. In addition, the predictor
parameter is removed.
hist
is now the default tree method
Starting from 2.0, the hist
tree method will be the default. In previous versions, XGBoost chooses approx
or exact
depending on the input data and training environment. The new default can help XGBoost train models more efficiently and consistently. (#9320, #9353)
GPU-based approx tree method
There’s initial support for using the approx
tree method on GPU. The performance of the approx
is not yet well optimized but is feature complete except for the JVM packages. It can be accessed through the use of the parameter combination device="cuda", tree_method="approx"
. (#9414, #9399, #9478). Please note that the Scala-based Spark interface is not yet supported.
Optimize and bound the size of the histogram on CPU, to control memory footprint
XGBoost has a new parameter max_cached_hist_node
for users to limit the CPU cache size for histograms. It can help prevent XGBoost from caching histograms too aggressively. Without the cache, performance is likely to decrease. However, the size of the cache grows exponentially with the depth of the tree. The limit can be crucial when growing deep trees. In most cases, users need not configure this parameter as it does not affect the model’s accuracy. (#9455, #9441, #9440, #9427, #9400).
Along with the cache limit, XGBoost also reduces the memory usage of the hist
and approx
tree method on distributed systems by cutting the size of the cache by half. (#9433)
Improved external memory support
There is some exciting development around external memory support in XGBoost. It’s still an experimental feature, but the performance has been significantly improved with the default hist
tree method. We replaced the old file IO logic with memory map. In addition to performance, we have reduced CPU memory usage and added extensive documentation. Beginning from 2.0.0, we encourage users to try it with the hist
tree method when the memory saving by QuantileDMatrix
is not sufficient. (#9361, #9317, #9282, #9315, #8457)
Learning to rank
We created a brand-new implementation for the learning-to-rank task. With the latest version, XGBoost gained a set of new features for ranking task including:
- A new parameter
lambdarank_pair_method
for choosing the pair construction strategy. - A new parameter
lambdarank_num_pair_per_sample
for controlling the number of samples for each group. - An experimental implementation of unbiased learning-to-rank, which can be accessed using the
lambdarank_unbiased
parameter. - Support for custom gain function with
NDCG
using thendcg_exp_gain
parameter. - Deterministic GPU computation for all objectives and metrics.
NDCG
is now the default objective function.- Improved performance of metrics using caches.
- Support scikit-learn utilities for
XGBRanker
. - Extensive documentation on how learning-to-rank works with XGBoost.
For more information, please see the tutorial. Related PRs: (#8771, #8692, #8783, #8789, #8790, #8859, #8887, #8893, #8906, #8931, #9075, #9015, #9381, #9336, #8822, #9222, #8984, #8785, #8786, #8768)
Automatically estimated intercept
In the previous version, base_score
was a constant that could be set as a training parameter. In the new version, XGBoost can automatically estimate this parameter based on input labels for optimal accuracy. (#8539, #8498, #8272, #8793, #8607)
Quantile regression
The XGBoost algorithm now supports quantile regression, which involves minimizing the quantile loss (also called “pinball loss”). Furthermore, XGBoost allows for training with multiple target quantiles simultaneously with one tree per quantile. (#8775, #8761, #8760, #8758, #8750)
L1 and Quantile regression now supports learning rate
Both objectives use adaptive trees due to the lack of proper Hessian values. In the new version, XGBoost can scale the leaf value with the learning rate accordingly. (#8866)
Export cut value
Using the Python or the C package, users can export the quantile values (not to be confused with quantile regression) used for the hist
tree method. (#9356)
column-based split and federated learning
We made progress on column-based split for federated learning. In 2.0, both approx
, hist
, and hist
with vector leaf can work with column-based data split, along with support for vertical federated learning. Work on GPU support is still on-going, stay tuned. (#8576, #8468, #8442, #8847, #8811, #8985, #8623, #8568, #8828, #8932, #9081, #9102, #9103, #9124, #9120, #9367, #9370, #9343, #9171, #9346, #9270, #9244, #8494, #8434, #8742, #8804, #8710, #8676, #9020, #9002, #9058, #9037, #9018, #9295, #9006, #9300, #8765, #9365, #9060)
PySpark
After the initial introduction of the PySpark interface, it has gained some new features and optimizations in 2.0.
- GPU-based prediction. (#9292, #9542)
- Optimization for data initialization by avoiding the stack operation. (#9088)
- Support predict feature contribution. (#8633)
- Python typing support. (#9156, #9172, #9079, #8375)
use_gpu
is deprecated. Thedevice
parameter is preferred.- Update eval_metric validation to support list of strings (#8826)
- Improved logs for training (#9449)
- Maintenance, including refactoring and document updates (#8324, #8465, #8605, #9202, #9460, #9302, #8385, #8630, #8525, #8496)
- Fix for GPU setup. (#9495)
Other General New Features
Here’s a list of new features that don’t have their own section and yet are general to all language bindings.
- Use array interface for CSC matrix. This helps XGBoost to use a consistent number of threads and align the interface of the CSC matrix with other interfaces. In addition, memory usage is likely to decrease with CSC input thanks to on-the-fly type conversion. (#8672)
- CUDA compute 90 is now part of the default build.. (#9397)
Other General Optimization
These optimizations are general to all language bindings. For language-specific optimization, please visit the corresponding sections.
- Performance for input with
array_interface
on CPU (likenumpy
) is significantly improved. (#9090) - Some optimization with CUDA for data initialization. (#9199, #9209, #9144)
- Use the latest thrust policy to prevent synchronizing GPU devices. (#9212)
- XGBoost now uses a per-thread CUDA stream, which prevents synchronization with other streams. (#9416, #9396, #9413)
Notable breaking change
Other than the aforementioned change with the device
parameter, here’s a list of breaking changes affecting all packages.
- Users must specify the format for text input (#9077). However, we suggest using third-party data structures such as
numpy.ndarray
instead of relying on text inputs. See #9472 for more info.
Notable bug fixes
Some noteworthy bug fixes that are not related to specific language bindings are listed in this section.
- Some language environments use a different thread to perform garbage collection, which breaks the thread-local cache used in XGBoost. XGBoost 2.0 implements a new thread-safe cache using a light weight lock to replace the thread-local cache. (#8851)
- Fix model IO by clearing the prediction cache. (#8904)
inf
is checked during data construction. (#8911)- Preserve order of saved updaters configuration. Usually, this is not an issue unless the
updater
parameter is used instead of thetree_method
parameter (#9355) - Fix GPU memory allocation issue with categorical splits. (#9529)
- Handle escape sequence like
\t\n
in feature names for JSON model dump. (#9474) - Normalize file path for model IO and text input. This handles short paths on Windows and paths that contain
~
on Unix (#9463). In addition, all…