mlfoundations / open-diffusion Goto Github PK

Would be great to have (optional) model evaluation.
Possibilities:

• CLIP score (e.g. on a reference set of captions like the ones from Parti)
• FID, or inception distance in general where we could use other models like CLIP to extract features, as Inception is ImageNet specific
• Possibily the recent ImageReward https://arxiv.org/abs/2304.05977, which relies on a model trained on human rankings and is quite easy to use, they are also planning to make the ranking dataset bigger

It would be nice to be able to avoid the need of an internet connection as it is usually blocked in supercomputers.
Two things I observed that needs internet connection:

• wandb, can be solved by putting mode to offline in wandb.init, so that could be an option in the yaml config file (by default would be False)
• loading HF pre-trained models, that could be solved by using environment variables export TRANSFORMERS_CACHE=<cache_folder and export TRANSFORMERS_OFFLINE=1, so this just needs to be documented. In my case, even after using those, I still got the following:

[1] 2023-04-23 06:19:21 WARNING  'HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /CompVis/stable-diffusion-v1-4/resolve/main/scheduler/scheduler_config.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x14806d3bfbe0>, 'Connection to huggingface.co timed out. (connect timeout=10)'))' thrown while requesting HEAD https://huggingface.co/CompVis/stable-diffusion-v1-4/resolve/main/scheduler/scheduler_config.json

which happens at each validation/generation step, although it did not have any consequence, the job did not hang or anything, but I did not find a way to get rid of the message.

I might have missed it in the code, but I can't see whether we randomly drop the captions for classifier-free guidance (which is already used at inference).

Thanks @vramanuj for this really nice repo!

I started to experiment with it and had an issue with symlinks that are used to indicate the current pipeline folder.
In my case, the symlinks were dead because they are pointing to relative path, so resuming did not work.
Simply adding .absolute() to save_path in https://github.com/mlfoundations/open-diffusion/blob/main/train.py#L311
and https://github.com/mlfoundations/open-diffusion/blob/main/train.py#L509 makes it work fine.

Or did you do anything else that made it work anyway?