bentoml / diffusers-examples

API serving for your diffusers models

bentoml.diffusers examples

This repository hosts supplementary materials of the article Creating Stable Diffusion 2.0 Service With BentoML And Diffusers.

Prompt: Kawaii low poly grey American shorthair cat character, 3D isometric render, ambient occlusion, unity engine, lively color Negative prompt: low-res, blurry, mutation, deformed

Currently we have three examples:

• sd2/ contains a service with txt2img/ endpoint utilizing stabilityai/stable-diffusion-2
• sd2_mega/ contains a service with txt2img/ and img2img/ endpoints utilizing stabilityai/stable-diffusion-2 and diffusers' custom pipline
• anything_v3/ contains a service with txt2img/ endpoint utilizing Linaqruf/anything-v3.0

Prepare environment

We recommend running these services on a machine equipped with a Nvidia graphic card and CUDA Toolkit installed.

First let's prepare a virtual environment and install requried depedencies

python3 -m venv venv/ && source venv/bin/activate
pip install -U -r requirements.txt

Import models

You may need to authorize your huggingface account to download models, to do that, run:

pip install -U huggingface_hub
huggingface-cli login

then:

• to import stabilityai/stable-diffusion-2, run python3 import_model.py
• to import Linaqruf/anything-v3.0, run python3 import_anything_v3

Start the service and go on

After the model is imported, you can go into sd2/, sd2_mega or anything_v3 and follow the readme inside the folder to start the service and make a docker image for each service

diffusers-examples's Issues

enable_sequential_cpu_offload HuggingFace Diffusers error with sd2 example on T4 GPU

Hi, I was following this example https://modelserving.com/blog/creating-stable-diffusion-20-service-with-bentoml-and-diffusers

or this by git clone of this example repo https://github.com/bentoml/diffusers-examples/tree/main/sd2

which results in a simple service.py file like this:

import torch
from diffusers import StableDiffusionPipeline

import bentoml
from bentoml.io import Image, JSON, Multipart

bento_model = bentoml.diffusers.get("sd2:latest")
stable_diffusion_runner = bento_model.to_runner()

svc = bentoml.Service("stable_diffusion_v2", runners=[stable_diffusion_runner])

@svc.api(input=JSON(), output=Image())
def txt2img(input_data):
    images, _ = stable_diffusion_runner.run(**input_data)
    return images[0]

After bentoml serve service:svc --production I get the following error (happens also with another custom model that I tried). It seems to be related to enable_sequential_cpu_offload by HuggingFace.

[ERROR] [runner:sd2:1] Traceback (most recent call last):
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/starlette/routing.py", line 650, in startup
    handler()
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 303, in init_local
    raise e
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 293, in init_local
    self._set_handle(LocalRunnerRef)
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 139, in _set_handle
    runner_handle = handle_class(self, *args, **kwargs)
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 24, in __init__
    self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/bentoml/_internal/frameworks/diffusers.py", line 443, in __init__
    self.pipeline: diffusers.DiffusionPipeline = load_model(
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/bentoml/_internal/frameworks/diffusers.py", line 182, in load_model
    pipeline = pipeline.to(device_id)
  File "/home/be/miniconda3/envs/diffusers310/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 639, in to
    raise ValueError(
ValueError: It seems like you have activated sequential model offloading by calling `enable_sequential_cpu_offload`, but are now attempting to move the pipeline to GPU. This is not compatible with offloading. Please, move your pipeline `.to('cpu')` or consider removing the move altogether if you use sequential offloading.

As general info, it runs on a GCS VM instance with T4 GPU - could this be the issue?