Make TabNet Scikit Compatible about tabnet

[Optimox]

Hi @ottonemo,

I did not have a very close look at skorch, but it seems that it should be easy you use the networks defined in pytorch-tabnet and try to use them with skorch.

If you want to give it a try and share the results I'll be happy to have a look.

[JackyP]

This package is really cool. Have you guys considered https://github.com/skorch-dev/skorch? It might have a lot of what you need out of the box.

[Optimox]

Thx @JackyP, did not know about this package I'll have a look!

[ottonemo]

Thx @JackyP, did not know about this package I'll have a look!

Hey, did you ever think of substituting your base with skorch? It seems there is a huge overlap (callback design is basically the same, sklearn support out of the box and it looks to me that your data granularity/flow is very similar). It looks to me as though both projects could benefit from that.

[ottonemo]

@Optimox I took the census example from the README and ported it to skorch. The notebook is here (nbviewer).

I am sure that I missed some details but overall it was quite easy to let skorch handle the callback/sklearn/scoring/loading/saving and having a thin abstraction layer to implement the feature importances/explanation. I haven't tested regression, nor have I attempted to implement unsupervised pre-training but I'm quite certain that these are not more that much of an effort.

Of course I know my argument is already lost as the roc_auc score for the skorch example is 0.5 lower :)

[Optimox]

Well it looks great to me, not much simpler but it's great, if I had known about skorch since the beginning maybe all the code would be based on that.

Just to understand, starting from where we are now (with a lot of things already packages inside the library) what are the advantages of using skorch?

[ottonemo]

Well it looks great to me, not much simpler but it's great, if I had known about skorch since the beginning maybe all the code would be based on that.

Yes, it is remarkable how similar the layout to skorch is :)

Just to understand, starting from where we are now (with a lot of things already packages inside the library) what are the advantages of using skorch?

We are currently working on integrating automatic mixed precision, you would get the performance boost of something like this basically for free once it is implemented in skorch. Getting these quality-of-life features is a good point in itself, I think. Integration with TensorBoard is possible as well as experiment logging services like Neptune, Sacred and Wandb. It might also be helpful to have the ability to use other data types, such as pandas DataFrames as input types to your network which skorch supports. Or being able to pickle everything. Since we are rigorously watching out to implement the get_params/set_params protocol, compatibility with grid search implementations is given, sklearn's grid search as well as distributed grid search using dask. There's also a CLI interface that let's you create CLI training scripts for your models by automatically shaping model parameters to CLI parameters.

From a maintenance perspective it is probably desirable to have to maintain as less boiler-plate code as possible. Using skorch would give that to you at the risk/cost to depend on yet another library. Personally I think this is a comparatively small risk as the API is pretty stable (has been for a while now) and even in the event of all three developers stopping to contribute to skorch the only thing that happens for the foreseeable future would be no new features since compatibility is guaranteed by torch 1.0. But the risk of that dependency is also the chance of not being the only one with a specific problem and having a common core is often a good base for solving a shared problem quickly. We are quite happy about feedback / issues and PRs :)

[Optimox]

Yep I see, definitely agree.

It does seem quite a lot of work though, plus it would be a huge breaking change. Moreover, we haven't created enough unitary test (=None haha) only functional tests. I'd be scared to make the switch for now but when I have time I'll look closer. What happens for pretraining? multi-task? etc...

Also feel free to open a PR if you want to add an example of pytorch-tabnet skorch usage like the notebook you shared.

[ottonemo]

Yep I see, definitely agree.

It does seem quite a lot of work though, plus it would be a huge breaking change. Moreover, we haven't created enough unitary test (=None haha) only functional tests. I'd be scared to make the switch for now but when I have time I'll look closer. What happens for pretraining? multi-task? etc...

Such a transition is certainly not easy and it does not get easier, I agree. From what I can tell it is less work than I had anticipated but the uncertainty is definitely an issue. Sadly, I am not proficient enough with the underlying model to spot non-obvious problems and my time budget is quite limited at the moment but I am willing to assist where I can. So, when you do have tests and decide that you want to port to skorch (which is pretty well tested actually ;)) I could definitely be able to assist you.

Also feel free to open a PR if you want to add an example of pytorch-tabnet skorch usage like the notebook you shared.

Sure, will do :) I will keep the structure of the posted notebook and add some explanation/fluff around it if that is OK.

[kittpa]

Hey @ottonemo
I am currently also looking into using tabnet with skorch and was wondering if you have already made any further advancements? :)

[ottonemo]

Hey @ottonemo
I am currently also looking into using tabnet with skorch and was wondering if you have already made any further advancements? :)

Sorry, no significant advancements, the state is still very similar to the initial notebook I posted earlier in this thread (https://nbviewer.jupyter.org/gist/ottonemo/b49721b7a9a7634a027c43fcaf0a5014). May I ask what your motivation is to use skorch? Parameter search?

I hope I find the time to post a draft PR later this day with a notebook showcasing grid search.