GithubHelp home page GithubHelp logo

ssrsgjyd / neuraltexture Goto Github PK

View Code? Open in Web Editor NEW
263.0 8.0 36.0 33 KB

Unofficial implementation of the paper "Deferred Neural Rendering: Image Synthesis using Neural Textures" in Pytorch.

Python 100.00%

neuraltexture's Introduction

Neural Texture

This repository implements Deferred Neural Rendering: Image Synthesis using Neural Textures .

Requirements

  • Python 3.6+
    • argparse
    • nni
    • NumPy
    • Pillow
    • pytorch
    • tensorboardX
    • torchvision
    • tqdm

File Organization

The root directory contains several subdirectories and files:

dataset/ --- custom PyTorch Dataset classes for loading included data
model/ --- custom PyTorch Module classes
util.py --- useful procedures
render.py --- render using texture and U-Net
render_texture.py --- render from RGB texture or neural texture
train.py --- optimize texture and U-Net jointly
train_texture.py --- optimize only texture
train_unet.py --- optimize U-Net using pretrained 3-channel texture

How to Use

Set up Environment

Install python >= 3.6 and create an environment.

Install requirements:

pip install -r requirements.txt

Prepare Data

We need 3 folders of data:

  • /data/frame/ with video frames .png files
  • /data/uv/ with uv-map .npy files, each shaped (H, W, 2)
  • /data/extrinsics/ with normalized camera extrinsics in .npy files, each shaped (3)

Each frame corresponds to one uv map and corresponding camera extrinsic parameters. They are named sequentially, from 0000 to xxxx .

We demonstrate 2 ways to prepare data. One way is to render training data, the code is at https://github.com/A-Dying-Pig/OpenGL_NeuralTexture. The other way is to reconstruct from real scene, the code is at https://github.com/gerwang/InfiniTAM .

Configuration

Rename config_example.py as config.py and set the parameters for training and rendering.

Train Jointly

python train.py [--args]

Train Texture

python train_texture.py [--args]

Train U-Net

python train_unet.py [--args]

Render by Texture

python render_texture.py [--args]

Render by Texture and U-Net Jointly

python render.py [--args]

neuraltexture's People

Contributors

ssrsgjyd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

neuraltexture's Issues

camera extrinsics

What exactly are the camera extrinsics here?I think camera extrinsics are generally parameters matrix include rotation and translatio, which has 6 DoF. But in the paper, camera extrinsics parameters' shape is 3.

what are the outputs?

Hi, thanks for the code!
I just started to learn this paper with your code.
As far as I understand, there are two neural networks, and I trained them jointly using the train.py, with 410 basketball images.

When I run render.py with the trained model, there are something I don't understand on the outputted image. What are they and how can I erase them?

Thanks!

Getting clean results/Artifacts over output

I've been at this for a while tweaking config.py but I can't seem to crack the code of generating a clean, grid-free output. Does anyone have any suggestions?

A secondary point that you should note is the weird clipped gamma curve on Obama for some reason.
image

UV Atlas

Hello, thanks for the implementation.

I was wondering if you experimented with UV Atlas like specified in the paper? I couldn't find any information in the paper about how they used UV Atlas other than "uv-parameterization is computed based on the Microsoft uv-atlas generator".

Thanks.

RuntimeError: tensor size mismatch

When I run train.py, line 52 in pipeline.py: "x [:, 3:12,:,:] = x [:, 3:12,:,:] * basis [:,:]" reports an error:

RuntimeError: The size of tensor a (512) must match the size of tensor b (9) at non-singleton dimension 3

Then I print out the corresponding dimension.
basis.shape torch.Size([32, 9])
x [:, 3:12,:,:].shape torch.Size([32, 9, 512, 512])
What's the problem?

OpenGL_NeuralTexture

when i set enable_output = false,The screen can be displayed normally.
when i set enable_output = true,recompile and run the program.
Uv and Camera extrinsics are saved correctly
The screen is black, but the saved picture is incorrect. When using object_id = 0, the saved picture is basically all black. When object_id = 1, the saved picture is similar to noise

Does anyone know the possible cause? @SSRSGJYD

Creating UV input data

I've looked all over the internet for how to simply render a UV map or "texel-to-pixel mapping" and have found nothing. As mentioned in the README, OpenGL_NeuralTexture is an option, but I am hesitant to use that repo because it's kinda messy, most of it is hardcoded and inflexible/unusable, and I couldn't even compile it on Windows.

Is there another way to get the UV input data?

Spherical harmonics coefficients

Thank you for your repository!
In the utils.py file you provide the following code for the Spherical harmonics coefficients:
sh[0] = 1 / np.sqrt(4 * np.pi)
sh[1:4] = 2 * np.pi / 3 * np.sqrt(3 / (4 * np.pi))
sh[4] = np.pi / 8 * np.sqrt(5 / (4 * np.pi))
sh[5:8] = 3 * np.pi / 4 * np.sqrt(5 / (12 * np.pi))
sh[8] = 3 * np.pi / 8 * np.sqrt(5 / (12 * np.pi))

It seems that you use the first 9 harmonics as it is done in the paper. I found the following formulas and they are identical to yours except the coefficients:
image
Picture taken from http://www.cse.chalmers.se/~uffe/xjobb/Readings/GlobalIllumination/Spherical%20Harmonic%20Lighting%20-%20the%20gritty%20details.pdf
I tried to play with the math a bit but didn't get any insight anyway.
Is there some reasoning behind your coefficients? Thank you!

Different results every time

For some reason, I have been getting different results every time I run train.py and I have no idea as to why. It has stopped me from getting any meaningful results.

Animation Synthesis - aka Animated Texture - aka Deepfake

In the paper "Deferred Neural Rendering: Image Synthesis using Neural Textures", the paper this repo is based off of, they explain how they can use a generated UV map of a 3D model of a face, and the rgb of the face, to generate a "deepfaked" face output, with their Neural Rendering method. I understand this is possible with this repo for a single facial pose but I would like to use this, like in the demonstration, with an animated UV and texture so-to-speak, where the camera doesn't move but the model/UV and texture does.

I have been using another repo Face2face to generate the model and transfer the expressions with some custom code and it would be absolutely incredible if I can realistically neurally render it with this repo.

Is this possible?

(Even if it isn't, I kinda want to try anyway and see what I get just for fun)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.