LGBMClassifier gives non-deterministic outputs with very low AUC score compared to xgboost and catboost about lightgbm

[jmoralez]

Hey @sktin, thanks for using LightGBM. I took a quick look and it seems that the model starts overfitting very quickly with the default settings. XGBoost sets regularization by default

lambda [default=1, alias: reg_lambda] (ref)

and LightGBM doesn't, so setting reg_lambda=1 improves the score (~0.83 for me).

To make the results reproducible there's a deterministic parameter (docs)

deterministic, default = false, type = bool
used only with cpu device type
setting this to true should ensure the stable results when using the same data and the same parameters (and different num_threads)

In summary, you should be able to get a similar, consistent score as with the other models with the following:

test_classifier(LGBMClassifier(n_jobs=4,random_state=0,verbose=-1,reg_lambda=1,deterministic=True))

[sktin]

@jmoralez Thank you for the quick response.

I confirm that with reg_lambda=1, even without using the deterministic parameter, I am able to get "normal" and consistent AUC scores.

Out of curiosity, I set reg_lambda=0 in XGBClassifier and it returns consistent score > 0.8 albeit lower than the default setting of reg_lambda=1. As far as the original issue for LGBMClassifier is concerned, I think it can be closed.