Struggling to get model to fit - Help Wanted about tabnet

[chadbreece]

Also, do you have any pointers for hyperparameters while scaling up? The full dataset I am trying to run is ~40M rows, I'm trying to tune the hyperparameters on this 10% sample before applying those parameters to the full dataset.

[Optimox]

Your train/val plot looks suspicious to me: that is strange that train and valid have the exact same scores at every epochs.

Have you tried a learning decay?

[chadbreece]

I'm using OneCycleLR right now, open to suggestions to try (scheduler or lr values) and I can follow up with results here.

Also, I have been using log cosh loss as my objective, MASE as my eval. My regression target is heavily right skewed so I recently tried RMSLE but that didn't change the dynamic you see above.

Here is an example where I trained with log cosh loss as my objective and use MAE as my eval (ignore the legend), this is a bit better but still pretty volatile.

[chadbreece]

I tried reducing the number of epochs and pct_start in OneCycleLR and got the following

Much more stable but still not seeing the training MASE getting much better than validation.

More playing yielded more of the same. XGBoost is often able to get down to <0.4 MASE but I can't seem to get tabnet below ~0.45

[Optimox]

A large batch size often plays the role of a regularization method because of the batch norm used during training. At the cost of a longer training time you can try to significantly lower batch_size and virtual_batch_size (like 64). I'm not sure you'll get better validation performance but you should be able to see some overfitting.

[chadbreece]

Doing this, I still was unable to get the model to overfit... Are there any other hyperparameters I should be looking to change to help this?

[Optimox]

larger n_d, n_a, larger number of steps: larger model capacity should enable overfitting capacity.