blunde1 / agtboost Goto Github PK
View Code? Open in Web Editor NEWAdaptive and automatic gradient boosting computations.
License: MIT License
Adaptive and automatic gradient boosting computations.
License: MIT License
Both xgboost
and lightgbm
implements monotone constraints.
This is important to practitioners.
An introduction to a possible implementation is given here
The node::split_information()
should be easy to paralellize.
I was trying to predict a binary variable specifying the loss_function as "logloss", but instead of stopping early, it seems the boosting rounds continue even while improvement stops. I expected training on the iris dataset to be very quick (like with loss_function "mse") but it just keeps going. The same issue is happening with other loss functions (similar issue with "poisson") except "mse" which performs as expected. The example can be reproduced as below.
Use testthat
As a minimum, write tests for
gbt.save()
and gbt.load()
gbt.train()
predict()
Currently, only loss_function=count::auto
returns E[y|x] by default.
It uses code
if(type %in% c("", "response")){
# predict mean
res <- object$predict(newdata)
res <- exp(res)
}else if(type == "link_response"){
# predict response on log (link) level
res <- object$predict(newdata)
}
Implement this for ordinary loss functions in R/gbt.pred
The CRAN policy contains
- Packages should not attempt to disable compiler diagnostics, nor to
remove other diagnostic information such as symbols in shared objects.
yet packages
MatchIt agtboost briskaR kgrams matrixprofiler prospectr resemble
strucchangeRcpp
attempt to do so.
Do correct before 2021-11-26 to safely retain your package on CRAN, and
as this will require manual check, do allow time for the CRAN
submissions team to do so.
Suggestions:
Makevars
Implement Shapley-values to explain model predictions.
See vignett for https://github.com/NorskRegnesentral/shapr
The most important parameters of other tree-boosting libraries should be possible to obtain from an agtboost
ensemble.
Possible parameters include
Usefulness
Implementations such as xgboost
are very robust, but time-consuming to tune.
It might be desirable to have an xgboost
model in production, but where parameters are trained from agtboost
.
I might be doing something wrong, maybe there are ways to make aGTBoost faster. As far as the AUC, this is just one dataset, I know.
Anyway, you might wanna look at this: https://github.com/szilard/GBM-perf
Currently agtboost
only supports a purely numerical design-matrix.
Consider letting agtboost
handle categorical features (factor
variable in R) internally.
Benefits
y~x1+x2+...
Downsides
Secret manuscript will arrive on arxiv first
In principle:
Should be possible with the Eigen sparse matrix class + R Matrix package and possible RcppModules to return pointer to C++ model object.
See xgboost implementation.
This would be easy if gradient-calculations was on the R-side.
But perhaps harder since agtboost
gradient calculations happen on R-side.
Suggestions
The following code:
library(agtboost)
data(caravan.train, package = "agtboost")
train <- caravan.train
gbt.train(train$y, train$x, loss_function="logloss", verbose=10, nrounds=500, learning_rate=0.1)
gives the following output:
it: 1 | n-leaves: 167 | tr loss: 0.1703 | gen loss: 0.1686
it: 10 | n-leaves: 80 | tr loss: 0.1073 | gen loss: 0.1041
it: 20 | n-leaves: 6 | tr loss: 0.05831 | gen loss: 0.0536
it: 30 | n-leaves: 432 | tr loss: 0.0302 | gen loss: 0.02553
it: 40 | n-leaves: 411 | tr loss: 0.01981 | gen loss: 0.01515
it: 50 | n-leaves: 433 | tr loss: 0.01582 | gen loss: 0.01117
it: 60 | n-leaves: 30 | tr loss: 0.01389 | gen loss: 0.009244
it: 70 | n-leaves: 434 | tr loss: 0.01263 | gen loss: 0.007985
it: 80 | n-leaves: 4 | tr loss: 0.01221 | gen loss: 0.007566
it: 90 | n-leaves: 2 | tr loss: 0.01183 | gen loss: 0.007186
it: 100 | n-leaves: 217 | tr loss: 0.01156 | gen loss: 0.006912
it: 110 | n-leaves: 473 | tr loss: 0.01138 | gen loss: 0.00674
it: 120 | n-leaves: 496 | tr loss: 0.01129 | gen loss: 0.006649
it: 130 | n-leaves: 252 | tr loss: 0.01125 | gen loss: 0.006602
it: 140 | n-leaves: 475 | tr loss: 0.01122 | gen loss: 0.006576
it: 150 | n-leaves: 485 | tr loss: 0.01121 | gen loss: 0.006563
it: 160 | n-leaves: 276 | tr loss: 0.0112 | gen loss: 0.006554
it: 170 | n-leaves: 276 | tr loss: 0.01119 | gen loss: 0.00655
it: 180 | n-leaves: 276 | tr loss: 0.01119 | gen loss: 0.006547
it: 190 | n-leaves: 276 | tr loss: 0.01119 | gen loss: 0.006546
it: 200 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006545
it: 210 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006545
it: 220 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 230 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 240 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 250 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 260 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 270 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 280 | n-leaves: 281 | tr loss: 0.01119 | gen loss: 0.006544
it: 290 | n-leaves: 282 | tr loss: 0.01119 | gen loss: 0.006544
it: 300 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 310 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 320 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 330 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 340 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 350 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 360 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 370 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 380 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 390 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 400 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 410 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 420 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 430 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 440 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 450 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 460 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 470 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 480 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 490 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
it: 500 | n-leaves: 1 | tr loss: -nan | gen loss: -nan
And predict
outputs NaN predictions.
insert library(agtboost)
The following example with low-frequent data does not converge.
Here, EY = 0.1% given that exposure is 1 (it is Ez = 0.5 for training data, but 1.0 for test).
Seems to break at iteration 824
# Import library
library(agtboost)
# Reproducible
set.seed(123)
# Approximately similar n, with low-frequent events
# x: feature, EX = 1
# z: exposure
# y: response
n <- 100000
average_frequendy <- 0.001
xtr <- as.matrix(runif(n, 0, 2)) # EX = 1
ztr <- runif(n)
ytr <- rpois(n, ztr* average_frequendy * xtr)
xte <- as.matrix(runif(n, 0, 2))
yte <- rpois(n, average_frequendy * xte) # test for z=1
# Plot data, ensure it looks appropriate
par(mfrow=c(1,2))
plot_sample <- sample(n, 10000)
plot(xte[plot_sample,], yte[plot_sample])
plot((xtr*ztr)[plot_sample,], ytr[plot_sample])
par(mfrow=c(1,1))
# Train agtb-model
agtb <- gbt.train(ytr, xtr, loss_function = "poisson", offset=log(ztr), verbose=1)
# Compare null-model and agtb-model on test-data
-sum(dpois(yte, mean(ytr), log=TRUE))
# agtb model
-sum(dpois(yte, predict(agtb, xte), log=TRUE))
# Inspect predictions
plot(xte[plot_sample,], yte[plot_sample])
points(xte[plot_sample], predict(agtb, xte)[plot_sample], col=2)
I think I have detected a memory leak in gbt.load. Could be I'm failing to understand how R handles memory (and the imperfections of that process), but maybe adding a manual garbage collection in the C subroutine could be the way to solve this?
Minimal replication:
x <- runif(50000, 0, 10)
y <- rnorm(50000, x, 1)
mod <- gbt.train(y, as.matrix(x))
gbt.save(mod, 'gbt_model.gbt')
Memory usage (by the R process, not the objects in R workspace) increments on my system by roughly 1mb every time the below line is run:
mod <- gbt.load('gbt_model.gbt')
Secret manuscript-recipe coming first on arxiv
When offset
is nonzero, boosting from the average of y
(optimal with zero offset) or from zero is sub-optimal.
Thus: Provide an optimal initial prediction when offset
is nonzero:
An optimization algorithm starting from the average is natural, unless no result is available.
Explore if g
and h
may be used for internal optimizer: Will be able to solve convex problems.
In this case, it is natural to create an "initial_boosting_prediction.h" module.
Secret manuscript will be uploaded to arxiv
This has an importance for many applications, but perhaps especially for Poisson-regression where observations have been observed at different time-lengths/fractions of total-time.
pred = g^{-1}(F(x) + offset)
Overall, codebase, especially on the C++ side, needs cleanup in terms of
Major revision would include removing different variants of count-regression except for ordinary Poissong with log-link.
The agtboost
model class contains the type of loss-function as an std::string
type attribute. This is problematic (for memory and garbage collection reasons) when working with a model object from the R side. See e.g. the following comment #55 (comment).
The model could and should consist of entirely numerical attributes.
There are no problems for the information criterion in regards to implementing a fast histogram-based algorithm.
This should make the sorting which is nlog(n)
be linear in n
.
The human-readable text-serialization is one of the most expensive ways to serialize. See the comment in #55 (comment).
Take a look at more efficient and robust ways of serialization.
Secret manuscript will be uploaded to arxiv
agtboost
does not scale well with large n
and m
.
In principle, it should scale as well as e.g. xgboost
with algorithm set to exact
.
There also seems to be problems with memory-consumption for large datasets.
Goal
Make agtboost
scale well in n
and m
.
Suspission
I suspect this is due to the re-declarations/initializations of u_store
and u
vectors and cir
calculations in split_information()
.
The model object cannot be saved by R-default save()
function as .RData
.
A custom save function must be created, say gbt.save_gbt_model()
that saves the model
as a binary file. Complementary gbt.load_gbt_model()
must also be created.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.