Issue compiling in docker - No CUDA runtime is found about gptq-for-llama

[TheTerrasque]

@fibulwinter is correct, the problem was GPU not being available during "docker build". Building the extension when running the container works fine.

[fibulwinter]

CUDA is not available on Docker build time without tricks: https://discuss.huggingface.co/t/how-to-deal-with-no-gpu-during-docker-build-time/28544

[NanoCode012]

I finally built a copy of this repo. Here was my setup.

1. docker pull nvidia/cuda:11.7.1-devel-ubuntu22.04 (Must be 11.7)

2. Start it up then using Nvidia-docker

3. Run

   apt update && apt upgrade -y
   apt install software-properties-common -y
   add-apt-repository ppa:deadsnakes/ppa
   apt install python3.10 -y
   apt install python3-pip -y

4. pip3 install torch

5. Run

   cd llama_model
   python3 setup_cuda.py install

[TheTerrasque]

I finally built a copy of this repo. Here was my setup.

1. docker pull nvidia/cuda:11.7.1-devel-ubuntu22.04 (Must be 11.7)
2. Start it up then using Nvidia-docker
3. Run

   apt update && apt upgrade -y
   apt install software-properties-common -y
   add-apt-repository ppa:deadsnakes/ppa
   apt install python3.10 -y
   apt install python3-pip -y

4. pip3 install torch
5. Run

   cd llama_model
   python3 setup_cuda.py install

Was all that part of docker build, or did some parts happen in docker run / exec?

[NanoCode012]

Hey @TheTerrasque , I was just testing to find the correct config, so I didn't put it in a Dockerfile. I just ran them manually using bash cli. You can definitely put them within the Dockerfile though.