davidwyand / unreal-diffusion

Stable Diffusion for Unreal Engine

Unreal Diffusion : A Stable Diffusion Unreal Engine Implementation

This is a fork of invoke-ai/InvokeAI, a popular SD implementation with great features and GUI. It provides a streamlined GUI inside Unreal Engine to easily generate game-ready textures.

Quick links: [Discord Server]

Table of Contents

1. Installation
2. Features
3. Latest Changes
4. Troubleshooting
5. Support

Installation

This fork is supported across multiple platforms. You can find individual installation instructions below.

• [1] Follow the install instructions at invoke-ai/InvokeAI and get that running normally

• [2] Copy the Unreal/Unreal_Diffusion folder into your unreal project's content folder

• [3] Go to Project Settings in unreal and then plugin/Python

• [4] Navigate and add your content/Unreal_Diffusion/Python folder to the "Additional Paths" list

• [4.1] Also add /Lib/site-packages from your python 3 install location on your computer like in the image below

• [5] (Optional) Go to Editor Preferences in Unreal and then "Loading & Saving"

• [5.1] Add the "/outputs/img-samples/unreal-diffusion" to the Directories to monitor list to automatically import the generated images to unreal. You can map these to /Game/Unreal_Diffusion/Output as in the image below

Running the plugin

First switch to the correct folder and conda environment and then start the server. It should say "Point your browser at xxxxxxx" if launched succesfully

(base) conda activate invokeai

(invokeai) cd C:\AI\Unreal-Diffusion (example, depending on where you cloned the repo)

(invokeai) python scripts/legacy_api.py --web --port 3333 -o outputs/img-samples/unreal-diffusion

Back in unreal, go to the Unreal Diffusion folder and find the "Unreal_Diffusion" Editor Utility Widget, right-click that and press "Run Editor Utility Widget as in the image below. This should start the UI which looks like this and you are ready to start generating images/textures

Features

Major Features

Unreal UI

• Realtime - toggle that will generate the images without freezing unreal, but may generate slower. Off by default
• Seamless - Forces SD to generate seamless images. Sometimes you could try to have this off and instead type seamless in the prompt

Most docs here link to InvokeAI's as they are very detailed and show usecases and syntax

• Prompts and Prompt Blending

• Negative Prompts

• Image To Image(WIP)

• Upscaling(WIP)

• Variations(WIP)

Latest Changes

• v0.5.0 (23 October 2022)
  • initial version with prompt/negative prompts, cfg, steps features

Troubleshooting

If the images won't generate due to VRAM, try lowering the resolution or temporarily load an empty level in unreal to generate images.

Contributing

Anyone who wishes to contribute to this project, whether documentation, features, bug fixes, code cleanup, testing, or code reviews, is very much encouraged to do so. If you are unfamiliar with how to contribute to GitHub projects, here is a Getting Started Guide.

A full set of contribution guidelines, along with templates, are in progress, but for now the most important thing is to make your pull request against the "development" branch, and not against "main". This will help keep public breakage to a minimum and will allow you to propose more radical changes.

Contributors

This fork is at the moment handled by me, with the main features/backend API coming from InvokeAI.

Support

For support, please use this repository's GitHub Issues tracking service. Feel free to send me an email if you use and like the plugin.

Original portions of the software are Copyright (c) 2022 Emil Eldstål