[Feature request] Examples for advanced machine learning tasks about trax

[afrozenator]

Thanks @cantwbr -- I like examples too, maybe as a first step outlining all the things we can do in a MD doc can be super-useful, i.e. the training commands that @lukaszkaiser wrote above, but more extensive.

The next thing would be good colabs that show training etc.

But, yes, please let us know where we can be better and more importantly what would help you more.

But note that we don't do datasets at all, those are still sourced from TFDS or T2T

[lukaszkaiser]

Thanks for the suggestion @cantwbr !

We have a number of working examples in Trax -- you can see under configs/ a bunch of more advanced models that work. For example, you can train Transformer to translate with

python -m trax.trainer --config_file=$PWD/trax/configs/transformer_wmt_ende_8gb_adafactor.gin

Or an image-generation newer model Reformer like this:

python -m trax.trainer --config_file=$PWD/trax/configs/reformer_imagenet64.gin

There are a few things though that we don't know and it'd be very helpful to hear from you:

• are bash commands ok or is it better to provide colabs (ipython notebooks)?
• should we focus on providing pre-trained models too? (it slows us down, but if it's important...)
• what's the best way to group or present these models?

Please let us know what you think and what would be most helpful!

[elfprince13]

I don't really care about bash vs colabs, but it would be great to have the examples integrated into some API documentation showing the main entry points for different models and layers.

[mprithiv]

Thanks for the suggestion @cantwbr !

We have a number of working examples in Trax -- you can see under configs/ a bunch of more advanced models that work. For example, you can train Transformer to translate with

python -m trax.trainer --config_file=$PWD/trax/configs/transformer_wmt_ende_8gb_adafactor.gin

Or an image-generation newer model Reformer like this:

python -m trax.trainer --config_file=$PWD/trax/configs/reformer_imagenet64.gin

There are a few things though that we don't know and it'd be very helpful to hear from you:

• are bash commands ok or is it better to provide colabs (ipython notebooks)?
• should we focus on providing pre-trained models too? (it slows us down, but if it's important...)
• what's the best way to group or present these models?

Please let us know what you think and what would be most helpful!

Personally, I wouldn't bother having a pre-trained model (but might be helpful for others). I would assume

python -m trax.trainer --config_file=$PWD/trax/configs/reformer_wmt_ende.gin

can be used to train reformer for translation. A guide about how to plug reformer for NMT task would be very helpful

[lhk]

Having pretrained models would be extremely useful.
I have a small hobby project for which I would like to host a translation API.
This would be no problem, but I don't have the GPU capacity to train the models myself.

[ElhamSol]

to run python -m trax.trainer --config_file=$PWD/trax/configs/mlp_mnist.gin
I get the error :

File "/home/elham/gin-config/trax/models/research/bert.py", line 20, in <module>
    from tensorflow.train import load_checkpoint
ModuleNotFoundError: No module named 'tensorflow.train

any ideas?
I already have tensorflow installed

update:
resolved by editing bert.py
I changed two lines:
from tensorflow.train import load_checkpoint >>> import tensorflow
ckpt = load_checkpoint(self.init_checkpoint) >> ckpt = tensorflow.train.load_checkpoint(self.init_checkpoint)

Still don't understand why this should be a problem

[lukaszkaiser]

These lines are already changed at head, I hope it works now :). Feel free to reopen or make a new issue of course if there are more problems.