henzler / neuralmaterial

Generative Modelling of BRDF Textures from Flash Images [SIGGRAPH Asia, 2021]

Neural Material

Official code repository for the paper:

Generative Modelling of BRDF Textures from Flash Images [SIGGRAPH Asia, 2021]

Henzler, Deschaintre, J. Mitra, Ritschel

[Paper] [Project page]

Data

The dataset is stored under flash_images and contains 306 train folders and 116 test folders (including images from Aitalla et al).

Install dependencies

conda env create --name neuralmaterial --file=environment.yml

Training

For training please run

python scripts/train.py

The default config is located at config/config_default.yaml.

Inference

Note, <test_id> is a relative path in the flash_images/test and <weight_name> is exptected to be located in the trainings/<run_name> folder.

Synthesis

In order to synthesise given flash images located in the test folder please run

python scripts/test.py --model <model_path> --test_image_id <test_id> --finetune <bool>

Interpolation

Given two images for interpolation please run

python scripts/interpolate.py --model <model_path> --weights1 <weight_name1> --weights2 <weight_name2> --test_image_id1 <test_id1> --test_image_id2 <test_id2>

If you would like to use fine-tuned weights please run the scripts/test.py command above in order to retrieve them.

Examples

Run the file run_examples.sh to synthesise / interpolate a few examples.

Citation

If you find our work useful in your research, please cite:

@article{henzler2021neuralmaterial,
  title={Generative Modelling of BRDF Textures from Flash Images},
  author={Henzler, Philipp and Deschaintre, Valentin and Mitra, Niloy J and Ritschel, Tobias},
  journal={ACM Trans Graph (Proc. SIGGRAPH Asia)},
  year={2021},
  volume={40},
  number={6},
}

Contact

If you have any questions, please email Philipp Henzler at [email protected].

neuralmaterial's Issues

ModuleNotFoundError: No module named 'lib.core'

After calling trainer.fit(model, data) on Win10:

(neuralmaterial) E:\projects\ThirdParty\neuralmaterial>python scripts\train.py
Working directory : E:\projects\ThirdParty\neuralmaterial\trainings\Neuralmaterial
[Data] 25000 train samples
[Data] 25000 val samples
[INFO] Checkpoint loaded.
[INFO] Metadata loaded.

[Model]
decoder | params: 2,778,214
encoder | params: 25,819,304
loss | params: 20,024,384
renderer | params: 0
rotation_encoder | params: 188
total params: 48,622,090

Epoch: 13 | train step: 0%| | 0/25000 [00:00<?, ?it/s]Traceback (most recent call last):
File "", line 1, in
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\runpy.py", line 268, in run_path
return _run_module_code(code, init_globals, run_name,
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "C:\Users\onepride\Anaconda3\envs\neuralmaterial2\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "E:\projects\ThirdParty\neuralmaterial\scripts\train.py", line 8, in
from lib.core.trainer import Trainer
ModuleNotFoundError: No module named 'lib.core'

error loading the pretrained model

Hi,

I get this error when I try to test the model:
"_pickle.UnpicklingError: invalid load key, 'v'."
I think it might have something to do with the model being uploaded to github, could you please upload it to google drive? @henzler

Thanks

Scripts do not execute after installation

None of the scripts execute after installation as per the instructions. This issue is apparently due to mismatches in dependencies.

Upon first execution, I got :
'ModuleNotFoundError: No module named 'tensorboard''

After installing tensorboard via pip, I got:
'ModuleNotFoundError: No module named 'kornia''

After installing kornia via pip, I got:
_op = torch._C.jit_get_operation(qualified_op_name)
RuntimeError: No such operator image::read_file

Installed on Ubuntu 18.04 and 20.04 on a gcp cloud VM with a T4 GPU without success.

Could you please review that the installation instructions are accurate?

Thanks

Distorted BRDF maps

OS: Linux
GPU: Tesla K80 (Kepler)

When running test.py, the reconstructed render seems to come out fine, but the roughness and specular BRDF maps appear to be distorted. The normal map on this particular example seems to be correct, but a few others gave me pause. Running on CPU doesn't seem to make a difference.

Error loading images when running run_examples.sh

Traceback (most recent call last):
File "scripts/test.py", line 58, in
image = tfm(io.read_image(image_dirs[0]))[None]
File "/home/galem/anaconda3/envs/neuralmaterial/lib/python3.8/site-packages/torchvision/io/image.py", line 230, in read_image
return decode_image(data)
File "/home/galem/anaconda3/envs/neuralmaterial/lib/python3.8/site-packages/torchvision/io/image.py", line 211, in decode_image
output = torch.ops.image.decode_image(input)
RuntimeError: Unsupported image file. Only jpeg and png are currently supported.