sxela / warpfusion

WarpFusion

WarpFusion

WarpFusion

Latest public version:

• v0.21-AGPL
• 
• Release Notes

If you find a public version elsewhere, before running as admin/root, make sure to check it for malware by comparing it to the latest notebook in this repo.

Greatly inspired by Cameron Smith's neural-style-tf

Example videos

Example videos and settings

Guides made by users:

• 05.05.2023, v0.10 Video to AI Animation Tutorial For Beginners: Stable WarpFusion + Controlnet | MDMZ

• 11.05.2023, v0.11 How to use Stable Warp Fusion

• 13.05.2023, v0.8 Warp Fusion Local Install Guide (v0.8.6) with Diffusion Demonstration

• 14.05.2023, v0.12 Warp Fusion Alpha Masking Tutorial | Covers Both Auto-Masking and Custom Masking

• 23.05.2023, v0.12 STABLE WARPFUSION TUTORIAL - Colab Pro & Local Install

• 15.06.2023, v0.13 AI Animation out of Your Video: Stable Warpfusion Guide (Google Colab & Local Intallation)

• 17.06.2023, v0.14 Stable Warpfusion Tutorial: Turn Your Video to an AI Animation

• 21.06.2023, v0.14 Avoiding Common Problems with Stable Warpfusion

• 21.06.2023, v0.15 Warp Fusion: Step by Step Tutorial

• 04.07.2023, v0.15 Intense AI Video Maker (Stable WarpFusion Tutorial)

• 15.08.2023, v0.17 BEST Laptop for AI ( SDXL & Stable Warpfusion ) ft. RTX 4090 - Make AI Art FREE and FAST!

• 25.08.2023, ComfyWarp v0.1 WarpFusion: Warp and Consistency explanation in ComfyUI

• 2.09.2023, ComfyWarp v0.2 WarpFusion: ComfyWarp iteration 2.

• 3.09.2023, v0.16+ WarpFusion - Multiple Masked Prompts Guide

• 20.09.2023, v0.19 Warp Fusion Tutorial | Video to AI Video | Stable diffusion (Hindi)

• 10.10.2023, ComfyWarp v0.4 WarpFusion: ComfyWarp v0.4.2 (schedulers, flow_blend)

• 4.11.2023, ComfyWarp v0.5 WarpFusion: ComfyWarp v0.5 - FixedQueue

• 13.11.2023, v0.27 Stable WarpFusion v0.27 - Changelog

• 17.01.2024, v0.30 Stable WarpFusion v0.30 - Changelog

Local installation guide for Windows (venv)

Run once per notebook version (recommended)

1. Create a folder for WarpFusion. It's recommended to have a general folder for WarpFusion and subfolders for each version. Like C:\code\WarpFusion\0.16.11\ for version 0.16.11
2. Download install.bat and save it into your WarpFolder, C:\code\WarpFusion\0.16.11\ in this example.
3. Run install.bat. It will download and install python, git, and create a virtual python environment called "env" inside our folder and install dependencies, required to run the notebook and jupyter server for local colab. When git install window appears, use the default settings. The installation will contiinue after you install git.
4. Download run.bat and save it into your WarpFolder, C:\code\WarpFusion\0.16.11\ in this example.

Run to launch

1. Execute run.bat. It will activate the environment and start jupyter server.
2. After the server has launched, go to https://colab.research.google.com
3. Click File -> Upload Notebook and upload the *.ipynb file
4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
5. Select "connect to a local runtime" and paste the URL that will be generated below, which looks like "http://localhost:8888/?token=somenumbers"
6. Click "Connect" and CTRL+F9 to run all cells.

Local installation guide for Linux-Ubuntu 22.04 (venv):

Pre-requisites:

• Make sure that Ubuntu packages for CUDA toolkit and the latest NVIDIA utils are installed. Check using nvidia-smi command.

• ⚠️ Warning: Destructive Process Ahead ⚠️

  Clean Python Environment:

  • If you haven't followed best practices for Python virtual environments, you may want to clean your system.

  • Be warned, this is a destructive process and will remove all Python packages installed in the global environment.

    pip freeze > uninstall.txt
    pip uninstall -r uninstall.txt
    sudo pip freeze > uninstall.txt
    sudo pip uninstall -r uninstall.txt
    rm -rf ~/.cache

Installation Steps:

1. Choose Directory:

   • Open a terminal and navigate to your home directory or a directory of your choice.

   cd $HOME or cd ~

2. Clone Repository:

   • Clone the WarpFusion repository.

   git clone https://github.com/WarpFusion/WarpFusion.git

3. Navigate to Folder:

   • Enter the WarpFusion directory.

   cd WarpFusion

4. Run the Linux Installer:

   • Make the script executable and run it.

   chmod +x linux_install.sh
   ./linux_install.sh

   • Script will prompt you to enter a "version" to create your working folder, this can be any name you choose as at will append to "WarpFusion", ie: "WarpFusion0.23.11

Run to launch

1. Navigate to your WarpFusion(version) folder and execute the run script:

   cd $HOME/WarpFusion(version)
   ./run.sh

2. After the server has launched, go to https://colab.research.google.com
3. Click File -> Upload Notebook and upload the *.ipynb file
4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
5. Select "connect to a local runtime" and paste the URL that will be generated below, which looks like "http://localhost:8888/?token=somenumbers"
6. Click "Connect" and CTRL+F9 to run all cells.

Troubleshoot python virtual environment issues

• Delete your python virtual environment "warpenv" and re-run the running the script (backup your models, images and videos just in case).

  cd $HOME/WarpFusion(version)
  rm -rf warpenv

Docker install

Run once to install (and once per notebook version)

1. Create a folder for warp, for example d:\warp
2. Download Dockerfile and docker-compose.yml to d:\warp
3. Edit docker-compose.yml so that volumes point to your model, init_images, images_out folders that are outside of the warp folder. For example, d:\models\:/content/models will expose d:\models as /content/models to the notebook
4. Download and install docker from here - https://docs.docker.com/get-docker/
5. Run docker-compose up --build inside the warp folder.
6. Go to https://colab.research.google.com
7. Click File -> Upload Notebook and upload the *.ipynb file
8. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
9. Select "connect to a local runtime" and paste the token that was generated in your docker container, but leave the url as localhost. Should look like "http://localhost:8888/?token=somenumbers"
10. Click "Connect" and CTRL+F9 to run all cells.

Run to launch

1. Run docker-compose up inside the warp folder.
2. Go to https://colab.research.google.com
3. File -> open notebook -> open your previouslty uploaded notebook
4. Click on the dropdown menu near "Connect" or "Reconnect" button on the topright part of the interface.
5. Select "connect to a local runtime" and paste the token that was generated in your docker container, but leave the url as localhost. Should look like "http://localhost:8888/?token=somenumbers"
6. Click "Connect" and CTRL+F9 to run all cells.

Credits

This notebook uses:

Stable Diffusion by CompVis & StabilityAI
K-diffusion wrapper by Katherine Crowson
RAFT model by princeton-vl
Consistency Checking (legacy) from maua
Color correction from pengbo-learn
Auto brightness adjustment from progrockdiffusion

AUTOMATIC1111: weighted prompt keywords, lora, embeddings, attention hacks
Reconstructed noise - based on changes suggested by briansemrau

ControlNet
TemporalNet, Controlnet Face and lots of other controlnets (check model list)
BLIP by SalesForce
RobustVideoMatting (as external cli package)
CLIP
FreeU Hack
Experimental ffmpeg Deflicker
Dw pose estimator
SAMTrack Segment-and-Track-Anything (with cli my wrapper and edits)
ComfyUI: sdxl controlnet loaders, control loras
animatediff base
animatediff wrapper for compvis models from comfyui-animatediff
IP Adapters implementation from sd-webui-controlnet

DiscoDiffusion legacy credits:

Original notebook by Somnai, Adam Letts and lots of other awesome people!

Turbo feature by Chris Allen

Improvements to ability to run on local systems, Windows support, and dependency installation by HostsServer

Warp and custom model support by Alex Spirin

Citation

If you find this code useful for your research, please cite:

@misc{Spirin2022,
  author = {Spirin, Alex},
  title = {warpfusion},
  year = {2022},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/Sxela/WarpFusion}},
}

warpfusion's Issues

fix incorrect fallback on missing frame during frame correction

add xformers

fix color range error in flow preview

https://cdn.discordapp.com/attachments/973802253645381645/1042151394636996698/image.png

add verbosity option for prints

print norm stats (avg)

add sd v2 img2img depth map version

add alternative frame normalization

add dataloader when calc flow

add error for missing ffmpeg for local install

Suggestion for avoiding watermark-induced artifacts

I often get a watermark in some of my frames and if the camera is rotating, it can sometimes evolve into an object in the scene. A possible way to avoid this might be to crop the stylized image, expand it back to the original size and apply inpaint on the border around the frame. I'm not super well-versed in Python, though, so I'm not sure how to do it myself.

Code Running Issue

Hi, do you have normal code, which can be run simply from python, like a script? Because testing and using this from colab is not the best way, also it's really hard to improve it or change

fix masking for turbo

test patchmatch inpainting

wacatac.b ml Trojan

I just installed the latest version and my system found a Trojan in the files being installed.

Where's the local install and instructions for Mac??

Need instruction and install file for Mac M1/M2

Click File -> Upload Notebook and upload the *.ipynb file - cant find this file

i cannot find this file:

Click File -> Upload Notebook and upload the *.ipynb file

where is the notebook?

Collab

make templates use schedules

save model path to settings

check if flow saves all frames before cc (maybe proc pool isn't closed)

warp in latent space

test forward flow warping

test cond_fn slower in 5.24 vs 5.23

add warp to frame mapping

Make schedule (map) for frame-optical flow pairs

add negative prompts

add load from txt options for settings

test runwayml inpainting model with warp

speed up video export (multithreading)

add separate folder for mp4 outputs

add option to warp towards current diffusion input image, not raw init frame

Suggestions

Fill out the details, saying what WarpFusion is would be a good place to start. The videos can go below it.
Include a picture or two.
Put some tags in so people can find your repo.

use k noise

make cgf_scale scheduler / template

Unable to connect to the runtime

I have a GPU-server with Ubuntu 20.04. When i build and run container with docker-compose up, then i get something like http://localhost:8888/?token=41763aacd.. blah-blah. No any problems with run WarpFasion. But when i insert that link as Backend URL to "Local connection settings" in Google Colab i get error that:

Unable to connect to the runtime

Also somehow i cannot put external IP-address of my server instead of "localhost"
How do i solve this problem?

add warp multiplier

safeguard template output types (int for steps, etc)

add matching to any frames

add widget dynamic gui

Fix high CPU RAM usage during StableDiffusion model init

SD model takes a lot of cpu ram during initial load, that may not fit into colab 12gigs of CPU ram.
We can try loading it onto GPU, but it will still cause memory leak and overhead, resulting in lower max res

add inverse mask

add tooltips to gui

add masking video

add latent norm frame selection like for colormatch

Fix Youtube links on the README

Hi The youtube links are not working properly.
Github is seeing the as links to repositories.

fix init mask video extraction

add v2.1 768-v model

add sampler selector

add video analyzer for automatic scheduling

add scheduling of params

add xformers for windows