hfg-gmuend / zoomaker

Zoomaker - Friendly house keeping for your AI model zoo and related resources

Zoomaker - Friendly house keeping for your AI model zoo and related resources.

Zoomaker is a command-line tool that helps install AI models, git repositories and run scripts.

• single source of truth: all resources are neatly defined in the zoo.yaml file
• freeze versions: know exactly which revision of a resources is installed at any time
• only download once: optimize bandwidth and cache your models locally
• optimize disk usage: downloaded models are cached

😻 TL;DR

1. Install Zoomaker pip install zoomaker
2. Define your resources in the zoo.yaml file
3. Run zoomaker install to install them

📦 Installation

pip install zoomaker

🦁 zoo.yaml Examples

Example of the zoo.yaml of a Stable Diffusion project with the Automatic1111 image generator:

name: my-automatic1111-model-zoo
version: 1.0
description: Lorem ipsum
author: your name

resources:
  image_generator:
    - name: automatic1111
      src: https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
      type: git
      revision: 22bcc7be428c94e9408f589966c2040187245d81
      install_to: ./

  models:
    - name: v2-1_768-ema-pruned
      src: stabilityai/stable-diffusion-2-1/v2-1_768-ema-pruned.safetensors
      type: huggingface
      install_to: ./stable-diffusion-webui/models/Stable-diffusion/

`zoo.yaml` example long

name: my-automatic1111-model-zoo
version: 1.0
description: Lorem ipsum
author: your name

aliases:
  image_generator: &image_generator ./
  models: &models ./stable-diffusion-webui/models/Stable-diffusion/
  controlnet: &controlnet ./stable-diffusion-webui/models/ControlNet/
  embeddings: &embeddings ./stable-diffusion-webui/embeddings/
  extensions: &extensions ./stable-diffusion-webui/extensions/

resources:
  image_generator:
    - name: automatic1111
      src: https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
      type: git
      revision: 22bcc7be428c94e9408f589966c2040187245d81
      install_to: *image_generator

  models:
    - name: v1-5-pruned-emaonly
      src: runwayml/stable-diffusion-v1-5/v1-5-pruned-emaonly.safetensors
      type: huggingface
      install_to: *models

  controlnet:
    - name: control_sd15_canny
      src: lllyasviel/ControlNet/models/control_sd15_canny.pth
      type: huggingface
      install_to: *controlnet

  embeddings:
    - name: midjourney-style
      src: sd-concepts-library/midjourney-style/learned_embeds.bin
      type: huggingface
      install_to: *embeddings
      rename_to: midjourney-style.bin
    - name: moebius
      src: sd-concepts-library/moebius/learned_embeds.bin
      type: huggingface
      install_to: *embeddings
      rename_to: moebius.bin

  extensions:
    - name: sd-webui-tunnels
      src: https://github.com/Bing-su/sd-webui-tunnels.git
      type: git
      install_to: *extensions

`zoo.yaml` with script snippets

Here are a few examples of how to run scripts snippets from the zoo.yaml file. For example for starting the Automatic1111's webui, you could setup snippets like these and then run them with zoomaker run start_webui. All scripts are run from the root of the project, please adjust the paths accordingly.

scripts:
  start_webui: |
    cd .\stable-diffusion-webui && call webui.bat

scripts:
  start_webui: |
    conda activate automatic1111
    cd /home/$(whoami)/stable-diffusion-webui/
    ./webui.sh --xformers --no-half

`zoo.yaml` with web download

resources:
  models:
    - name: analog-diffusion-v1
      src: https://civitai.com/api/download/models/1344
      type: download
      install_to: ./stable-diffusion-webui/models/Stable-diffusion/
      rename_to: analog-diffusion-v1.safetensors

Please note: The resource type: download can be seen as the last resort. Existing web downloads are skipped, but no other caching. It is recommended to avoid web downloads :)

🧮 zoo.yaml Structure

Top level:

• name (mandatory)
• version, description, author, aliases (optional)
• resources (mandatory) : <group-name> : [] (array of resources)
• scripts (optional) : <script-name>

Resource:

• name, src, type, install_to (mandatory)
• rename_to (optional)
• revision (optional), if none is defined the latest version from the main branch is downloaded
• type can either be git, huggingface or download

🧞 Zoomaker Commands

All commands are run from the root of the project, where also your zoo.yaml file is located.

Command	Action
zoomaker install	Installs resources as defined in zoo.yaml
zoomaker run <script_name>	Run CLI scripts as defined in zoo.yaml
zoomaker --help	Get help using the Zoomaker CLI
zoomaker --version	Show current Zoomaker version

🤗 Hugging Face Access Token and Custom Cache Location

You might be asked for a Hugging Face Access Token during zoomaker install. Some resources on Hugging Face require accepting the terms of use of the model. You can set your access token by running this command in a terminal. The command huggingface-cli is automatically shipped alongside zoomaker.

huggingface-cli login

You can specify a custom cache location by setting the HF_HOME environment variable. The default cache location is ~/.cache/huggingface/.

export HF_HOME=/path/to/your/cache
zoomaker install

🙏 Acknowledgements

• Most of the internal heavy lifting is done be the huggingface_hub library by Hugging Face. Thanks!
• "Zoomaker Safari Hacker Cat" cover image by Alia Tasler, based on this OpenMoji. Thanks!

zoomaker's Issues

error running tests on windows for RC 0.8

when running the tests on windows I get this error

python tests.py
.       downloading: 100%|███████████████████████████████████████████| 1.78k/1.78k [00:00<00:00, 1.78MiB/s]
.       downloading: 164kiB [00:00, 43.0MiB/s]
learned_embeds.bin: 100%|█████████████████████████████████████████████████████████████████| 3.82k/3.82k [00:00<?, ?B/s]
.E..
======================================================================
ERROR: test_install_huggingface_cached (__main__.ZoomakerTestCase.test_install_huggingface_cached)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\user\Documents\projects\zoomaker\test\tests.py", line 88, in test_install_huggingface_cached
    self.assertTrue(os.path.exists(filepath))
                    ^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: exists: path should be string, bytes, os.PathLike or integer, not NoneType

It looks like try_to_load_from_cache returns None which should mean that no cache is found according to the docs:

Returns:
Optional[str] or _CACHED_NO_EXIST:
Will return None if the file was not cached. Otherwise:
- The exact path to the cached file if it's found in the cache
- A special value _CACHED_NO_EXIST if the file does not exist at the given commit hash and this fact was
cached.

Expose via --cache-folder on CLI

https://gitlab.rlp.net/kitegg/hsm/jupyterhub-singleuser/-/issues/1

plz ignore (an issue to save assets)