keijiro / pix2pix

Real-time pix2pix implementation with Unity

Pix2Pix for Unity

This is an attempt to run pix2pix (image-to-image translation with deep neural network) in real time with Unity. It contains its own implementation of an inference engine, so it doesn't require installation of other neural network frameworks.

Sketch Pad demo

Sketch Pad is a demonstration that resembles the famous edges2cats demo but in real time. You can download a pre-built binary from the Releases page.

Demo video

System requirements

• Unity 2018.1
• Compute shader capability (DX11, Metal, Vulkan, etc.)

Although it's implemented in a platform agnostic fashion, many parts of it are optimized for NVIDIA GPU architectures. To run the Sketch Pad demo flawlessly, it's highly recomended to use a Windows system with GeForce GTX 1070 or greater.

How to use a trained model

This repository doesn't contain any trained model to save the bandwidth and storage quota. To run the example project on Unity Editor, download the pre-trained edges2cats model and copy it into Assets/StreamingAssets.

This implementation only supports the .pict weight data format which is used in Christopher Hesse's interactive demo. You can pick one of the pre-trained models or train your own model with using pix2pix-tensorflow. To export weight data from a checkpoint, please see the description in the export-checkpoint.py script.

pix2pix's Issues

Mobile Support

Hello Keijiro

Great implementation and optimization. Learning a lot about GPU compute by studying this code.

I am trying to get this to run on an iPhone X. I believe it could, just need to limit the kernel usage.

I might implement this myself and offer a drop down enum setting for mobile and desktop.

Have you any suggestions where to start in the code. I'll then make a pull request so you can see if it's worth it.

Regards

Unity3D 2019.2 / Rendering Issue

Hi Keijiro,

first of all thank you for your all your contributions!

I am using Unity3D 2019.2 on Windows 10 and the newest version of Pix2Pix does not work anymore. The rendering issue is there again. What I did so far:

1. I updated my Nvidia driver
2. I checked my .pict file if is in the StreamingAssets directory
3. I updated my PostProcessing plugin to 2.1.7 (I checked almost every version)
4. I rebooted my PC
5. I tried a completely different PC with same configuration

Since I was testing your previous version of the Pix2Pix Unity project, I checked the older version with Unity3D 2019.2 and it works like a charm. Sadly, the newest version is producing only this picture:

Model format

Hello keijiro,
I've tried to trained my own model by https://github.com/affinelayer/pix2pix-tensorflow, and after training I got some data like 'checkpoint' and 'graph.pbtxt'. However I could not find the .pict data.
What should I do next to get the .pict data?

Unity preprocessing pix2pix effect does not render as intended

Hi,

I have some issues with the rendering in unity that differs from rendering it using python script.
I'm using my own model.

Here is a test ( from pix2pix python script using the unity camera as inputs : )

Here is the same scene using the unity pix2pix model as preprocessing effects:

If you look closely, even if the python version is not train enough you can see difference between the sky, the wall and the floor. Whereas the Unity model blend all thing together.

I don't know how I can improve these render

I'm using Unity 2018.2.8.1f on Windows 10

Thanks in advance

Can some versions of Tensorflow cause problems?

Hello Keijiro,
about 2 months ago I trained my own model and exported a .PICT that worked fine with your plugin.
Recently I formatted my hard drive, reinstalled everything, trained a new model and exported a .PICT again, but it throws hundreds of errors when loading it with your plugin, this doesn't happen with the old .PICT.
I don't think the new model is corrupted or anything, it works well outside of Unity.
Does training with different versions of Tensorflow or pix2pix-tensorflow can affect the final results?
If so, which version/commit of Tensorflow/pix2pix-tensorflow do you suggest to use?
I used Tensorflow-gpu 1.8.0 for training on pix2pix-tensorflow (which I reset to april version with "git reset --hard d6f8e4ce00a1fd7a96a72ed17366bfcb207882c7"), the "export-checkpoint.py" file was missing from this april version, so I downloaded the last version for doing the export after the training.

Extracting edges from pictures with HED for free and offline

Hello Keijiro,
Is there a method to extract edges from pictures with Holistically-Nested Edge Detection ( https://github.com/phillipi/pix2pix#extracting-edges ) without your Google Coolaboratory solution here: https://twitter.com/_kzr/status/1043154594468122624 ?
I tried to do it offline with this ( https://github.com/s9xie/hed#batch-processing ) and got to the point where I run "batch_hed.py" ( https://github.com/phillipi/pix2pix/blob/master/scripts/edges/batch_hed.py ) and all my pictures are converted into .MAT files (MATLAB) files, the final step requires to run "PostprocessHED.m" ( https://github.com/phillipi/pix2pix/blob/master/scripts/edges/PostprocessHED.m ) to convert them into normal pictures, but instructions says that I need MATLAB to do it: "get the cpp file edgesNmsMex.cpp from https://raw.githubusercontent.com/pdollar/edges/master/private/edgesNmsMex.cpp and compile it in Matlab: mex edgesNmsMex.cpp You also need to download and install Piotr's Computer Vision Matlab Toolbox: https://pdollar.github.io/toolbox/"
Is there any method to do it for free? Matlab is a proprietary paid software, I noticed that your Coolaboratory is using Octave as a free alternative, how would I do it offline? I'm having trouble understanding how you managed to do it.
Any help is much appreciated, thank you very much for your kind attention!

Running with camera input or Spout

Hi Keijiro, excellent work as always. Was wondering how I could implement a game camera as an input via rendertexture for the pix2pix rather than a drawn image? similar to what were doing on your fb videos.

Model implementations

Hello Kejiro
Is it possible to use image model from civitai or huggingface website?
would love to see how community made models could work here.

on Android generate pure grey output

I have modified the UI from real time to a button submission and build onto an Android phone(samsung s9 Android 8.0). And I use Application.persistentDataPath to read the model .pict. It runs no error log on my Android phone now, but the output are always pure grey.

Unity Problem

Hi Keijiro

I wonder if I could ask you a couple of questions about this? I saw your experiments on twitter, feeding it a render of some geometry and I'm really interested to try some thing out with it myself.

Unfortunately though I can't even get the demo scene to run. Could that be a problem with my GPU? I have a Quadro K400 which is good for heavy geometry and so on but is terrible for GPU rendering [Redshift / Octane etc]. I do have Direct X [12].

This is a grab of my Unity - as you can see refresh rate is 0...

I tried the pre-built binary too and that did sort of run but not as you'd expect. Grab also attached.

Thanks very much for your time. It's fascinating work you're doing..

In case anyone is having StdLib.hlsl issues:

Replace:

#include "PostProcessing/Shaders/StdLib.hlsl"

With:

#include "Packages/com.unity.postprocessing/PostProcessing/Shaders/StdLib.hlsl"

Null Reference Exception on UseCommandBuffer

Here's the full error:

NullReferenceException: Object reference not set to an instance of an object Pix2Pix.GpuBackend.UseCommandBuffer (UnityEngine.Rendering.CommandBuffer buffer) (at Assets/Pix2Pix/GpuBackend.cs:68)

For the record, I'm on a Mac, using Unity 2017.3. Any ideas?

Output not as expected

https://imgur.com/a/XeEXZJC

This is on Windows 10, Nvidia 1080 Ti.

How to export .pict model files?

I wanted to try out your application with the CityScapes dataset (https://www.cityscapes-dataset.com/) but when I use the export python script, I get a human readable .bin file whilst the .pict file you use is in binary format. How do I convert the dataset into the format you use, or how to I export it properly to work with your application?