maxhalford / starboost

:star::rocket: Gradient boosting on steroids

Please check out the website if you're looking for the documentation!

What is this?

This is StarBoost, a Python library that implements gradient boosting. Gradient boosting is an efficient and popular machine learning algorithm used for supervised learning.

Doesn't scikit-learn already do that?

Indeed scikit-learn implements gradient boosting, but the only supported weak learner is a decision tree. In essence gradient boosting can be used with other weak learners than decision trees.

What about XGBoost/LightGBM/CatBoost?

The mentioned libraries are the state of the art of gradient boosting decision trees (GBRT). They implement a specific version of gradient boosting that is tailored to decision trees. StarBoost's purpose isn't to compete with them. Instead it's goal is to implement a generic gradient boosting algorithm that works with any weak learner.

A focus of StarBoost is to keep the code readable and commented, instead of obfuscating the algorithm under a pile of tangled code.

What's a weak learner?

A weak learner is any machine learning model that can learn from labeled data. It's called "weak" because it usually works better as part of an ensemble (such as gradient boosting). Examples are linear models, radial basis functions, decision trees, genetic programming, neural networks, etc. In theory you could even use gradient boosting as a weak learner.

Is it compatible with scikit-learn?

Yes, it is.

How do I install it?

Barring any weird Python setup, you simply have to run pip install starboost.

How do I use it?

The following snippet shows a very basic usage of StarBoost. Please check out the examples directory for comprehensive examples.

from sklearn import datasets
from sklearn import tree
import starboost as sb

X, y = datasets.load_boston(return_X_y=True)

model = sb.BoostingRegressor(
    base_estimator=tree.DecisionTreeRegressor(max_depth=3),
    n_estimators=30,
    learning_rate=0.1
)

model = model.fit(X, y)

y_pred = model.predict(X)

You can find the source code for running the benchmarks here.

What are you planning on doing next?

• Logging the progress
• Handling sample weights
• Implement more loss functions
• Make it faster
• Newton boosting (taking into account the information from the Hessian)
• Learning to rank

By the way, why is it called "StarBoost"?

As you might already know, in programming the star symbol * often refers to the concept of "everything". The idea is that StarBoost can be used with any weak learner, not just decision trees.

License

The MIT License (MIT). Please see the LICENSE file for more information.

starboost's Issues

base_estimator

Hi, is that I can change the base_estimator to another model, e.g. Linear regression ?

ValueError: non-broadcastable output operand with shape (200,) doesn't match the broadcast shape (200,200)

When trying to use starboost I am running into a problem in boosting.py line 124. There is mismatch between the shapes of y_pred[:,i] and direction. I think the line should be:

y_pred[:, i] += self.learning_rate * direction[:, i]