wpafbo79 / ai-docker Goto Github PK

CUDA and miniconda3 are common libraries/tools used by other projects. Put these in their own image to simplify adding new project images.

In order for images to build off each other, a common virtual environment directory is necessary.

Use /.venv/.

-oh i did ran that manually in the dir because reasons.. So maybe to do with that..

[Warning] One or more build-args [GIT_REPO] were not consumed
Successfully built c8ce71c6ca5b
Successfully tagged wpafbo79/cuda-with-prereqs:cuda-11.8.0_tini-0.19.0
Successfully tagged wpafbo79/cuda-with-prereqs:latest
Uploading...
The push refers to repository [docker.io/wpafbo79/cuda-with-prereqs]
7502adfe1a30: Preparing
781ffffa5ade: Preparing
22a11bc61346: Preparing
2fcd5b69d61b: Preparing
eb64737a07df: Preparing
0006e9e9490d: Waiting
40043f2eb50b: Waiting
6515074984c6: Waiting
denied: requested access to the resource is denied

I think i can just ignore it, but i just wanted you to know.

PS: How do you 'make' i always get: (Maybe i missing some tools?)

(base) root@linuxdesktop:~/docker/ai-docker-ET$ make
echo cuda-with-prereqs-images mini-cuda-base-images mini-cuda-with-prereqs-images real-esrgan-images gfpgan-images basujindal-stable-diffusion-images compvis-stable-diffusion-images invokeai-images
cuda-with-prereqs-images mini-cuda-base-images mini-cuda-with-prereqs-images real-esrgan-images gfpgan-images basujindal-stable-diffusion-images compvis-stable-diffusion-images invokeai-images
make -C cuda-with-prereqs/ image
make[1]: Entering directory '/home/root/docker/ai-docker-ET/cuda-with-prereqs'
./create-compose-file.sh > docker-compose.yml
./build.sh

./build.sh: 4: source: not found
./build.sh: 5: source: not found
./build.sh: 7: build: not found
make[1]: *** [Makefile:9: image] Error 127
make[1]: Leaving directory '/home/root/docker/ai-docker-ET/cuda-with-prereqs'
make: *** [Makefile:59: cuda-with-prereqs-images] Error 2

UPDATE:
Adding SHELL := /bin/bash in Makefiles works on my system.

In order to see what warnings/errors are generated during a build, save off the logs as files.

When creating a container, allow for an image version tag other than "latest" to be used.

Remove the DOCKER_BUILDKIT=0 line from the build script.

Instead of doing the package install for each project which results in slightly different layers that can't be shared, force the creation of a common layer by using an image.

Incorporate the CUDA toolkit into the image to speed up processing.

The current build system isn't sophisticated enough to properly detect when to rebuild an image. It either doesn't do it when it should, or does it when it shouldn't. There is too much care and feeding involved.

Create the image based on the instructions on the project website.

Create the image based on the instructions on the project website.

Use more sophisticated syntax to reduce duplication when adding new projects.

InvokeAI stores merged models in the models/merged_models/ directory.

Similar to the archive script, create a script to copy the files back to the volume.

Create an image based on the instructions on the project website.

Create the image based on the instructions on the project website.

Doing a git pull on a fixed commit will not work as intended and go to the head of the main/master branch. Extra steps to fetch all the commits and switch to the main branch will be necessary.

Cuda v12 images have been released: https://hub.docker.com/r/nvidia/cuda

Create image for base on instructions on the official website.

Set defaults for GIT_COMMIT and GIT_REPO for projects that don't set them. With strict options set in Bash, undefined variables crash the script.

Reference the project page for instructions and models.

This video has a good text file with the environment setup to copy.

When the archive directories get created, they are just the leaf folders, not the full path from the config. Creating the symbolic link needs to take that into account.

Update Git permissions of scripts so that only locally executable files are executable and other scripts are non-executable.

Create image based on instructions from the project website.

The current version is very old (circa 2019) and needs to be updated to something recent (e.g. 22.11.1).

Versions are listed here: https://repo.anaconda.com/miniconda/

Use the latest instructions from the project website.

Docker has Tini built-in starting with v1.13. It isn't necessary to explicitly include it.

For projects with releases, use the releases. Otherwise use particular Git commits.

Doing this will allow for images to remain usable OOTB. Development is so rapid that multiple projects break while fixing one.

• InvokeAI
• CompVis Stable Diffusion
• TencentARC GFPGAN
• Xintao's Real-ESRGAN
• Basu Jindal's Stable Diffusion
• AUTOMATIC1111's Stable Diffusion

Original project repo and Linux Docker repo.

The original project was written for Windows, so use the Docker fork.

Instead of going from folder to folder to build the Docker images, create a script to build the images in the correct order.

Updated the build scripts, Dockerfile, and the start script to use the config.sh file.

Be consistent and use apt-get in scripts.

Several option flags are used in scripts and should be avoided for volumes since the indices are turned into script flags and shouldn't collide with existing flags.

Avoid "b", "h", "n", and "v".

Allow for the preservation of ".previd" files when running make clean.

Create image based on instructions on project website.

Create image based on instructions on the official website.

Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: WSL environment detected but no adapters were found: unknown

Create the image based on the instructions on the project website.

Make it easier to read Dockerfiles by removing long Git repo strings and storing them in the config file. Rename the existing REPO to DOCKER_REPO to distinguish it from the newly added GIT_REPO.

Docker now mounts volumes to the symlink target instead of overriding the symlink. This makes the archived files hidden. To prevent Docker from hiding the archived files with mounted volumes an preserve file system size hard link all archived files.

To better ensure correct behavior and failure when a command fails, add the following options:

set -e
set -u
set -o pipefail

Ensure start.sh sources config.sh to allow for customization of behavior.