kai-46 / iron Goto Github PK

Hello! I tried to run IRON on my own dataset however it is not producing any viable results.
I assumed you were using NeRF++ for producing the cam_dict_norm.json, however there is very little detail on exactly how you processed the images obtained from camera.
Could you provide more details on the steps you took to process the images to create the train/test folders?

Thanks for your excellent work!
I use the xmen data you provoided, but the result is worse than the paper said.
The test_result as follow:

And the result of superman:

An error "index 0 is out of bounds for axis 1 with size 0" occurred while exporting the material.
vertices, face_vertices, texturecoords, face_texturecoords = loadmesh_and_checkuv(mesh_fpath, out_dir)
This code gets the shape of the variables texturecoords and face_texturecoords are both (0,0). Why？

Hi author,
According to your code, neus is used for stage 1 reconstruction. However, for each input dark field image, the lighting changes along with the camera. As far as I know, neus should only work in static-lighting scenarios. I wonder why neus works in your setting? Does it mean neus explain the dynamic flash light as view-dependent effects?

Hi Kai,
Appreciate for your another great work! The results of IRON look amazing!
However, I am confused about the "roughness range loss" mentioned in the paper. Why should we encourage the estimated roughness to stay below 0.5? What if we meet object with large roughness in custom data?
Looking forward to your advice.

Hi Kai!
I just use train_scene.sh with "xmen" data, but the result mesh in ./exp_iron_stage2/xmen/mesh_and_materials_50000 is too smooth:

this is the config file:
general {
base_exp_dir = ./exp_iron_stage1/CASE_NAME/
recording = [
./,
./models
]
}

dataset {
data_dir = ./datasets/Luanetal2021/CASE_NAME/train/
render_cameras_name = cameras_sphere.npz
object_cameras_name = cameras_sphere.npz
}

train {
learning_rate = 5e-4
learning_rate_alpha = 0.05
end_iter = 100001

batch_size = 1800
validate_resolution_level = 4
warm_up_end = 5000
anneal_end = 50000
use_white_bkgd = False

save_freq = 10000
val_freq = 2500
val_mesh_freq = 5000
report_freq = 100

igr_weight = 0.1
mask_weight = 0.0

}

model {
nerf {
D = 8,
d_in = 4,
d_in_view = 3,
W = 256,
multires = 10,
multires_view = 4,
output_ch = 4,
skips=[4],
use_viewdirs=True
}

sdf_network {
    d_out = 257
    d_in = 3
    d_hidden = 256
    n_layers = 8
    skip_in = [4]
    multires = 6
    bias = 0.5
    scale = 1.0
    geometric_init = True
    weight_norm = True
}

variance_network {
    init_val = 0.3
}

rendering_network {
    d_feature = 256
    mode = idr
    d_in = 9
    d_out = 3
    d_hidden = 256
    n_layers = 8
    skip_in = [4]
    weight_norm = True
    multires = 10
    multires_view = 4
    squeeze_out = True
}

neus_renderer {
    n_samples = 64
    n_importance = 64
    n_outside = 32
    up_sample_steps = 4     # 1 for simple coarse-to-fine sampling
    perturb = 1.0
}

}
Do you know why?

And I can't find the data "dragon" for training and testing in the dataset:

could you provide it for me to compare with you results?

Thanks for your excellent work!
I use the duck data from phySG you provoided, and I used the mask *.png in stage1, mask_weight=100.0, but the result of the stage2 is incorrect.
The logim result as follow:

Can you tell me where the problem might be?

Hi,
Thank you for a great contribution!
I am trying to import the created mesh.obj into blender, together with diffuse_albedo.png, specular_albedo.png, and roughness.png. I do this by creating a Principled BSDF material in blender and assigning the images to Base Color, Specular, and Roughness slots, respectively. The results are not what I expected. The images look dim, which I assume means that roughness is too high? On the other hand, the test-images rendered by python render_surface.py ... --render_all look good.
Any help is appreciated!

train_scene.sh drv/rabbit
Hello Wooden
Load data: Begin
Not using masks
image shape, mask shape: torch.Size([324, 768, 1024, 3]) torch.Size([324, 768, 1024, 3])
image pixel range: 0.0 1.0
Load data: End
0%| | 0/100001 [00:00<?, ?it/s]
Traceback (most recent call last):
File "render_volume.py", line 449, in
runner.train()
File "render_volume.py", line 127, in train
render_out = self.renderer.render(
File "/home/michael/iron/models/renderer.py", line 374, in render
ret_fine = self.render_core(
File "/home/michael/iron/models/renderer.py", line 233, in render_core
gradients = sdf_network.gradient(pts)
File "/home/michael/iron/models/fields.py", line 110, in gradient
gradients = torch.autograd.grad(
File "/home/michael/anaconda3/envs/iron/lib/python3.8/site-packages/torch/autograd/init.py", line 275, in grad
return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 5.80 GiB total capacity; 4.03 GiB already allocated; 118.56 MiB free; 4.08 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Wrote config file to ./exp_iron_stage2/drv/rabbit/args.txt
render_surface.py:256: DeprecationWarning: Starting with ImageIO v3 the behavior of this function will switch to that of iio.v3.imread. To keep the current behavior (and make this warning dissapear) use import imageio.v2 as imageio or call imageio.v2.imread directly.
im = imageio.imread(fpath).astype(np.float32) / 255.0
ic| fill_holes: False
handle_edges: True
is_training: True
args.inv_gamma_gt: False
0%| | 0/50001 [00:00<?, ?it/s]ic| args.out_dir: './exp_iron_stage2/drv/rabbit'
global_step: 0
loss.item(): 0.00573146715760231
img_loss.item(): 0.0
img_l2_loss.item(): 0.0
img_ssim_loss.item(): 0.0
eik_loss.item(): 0.00573146715760231
roughrange_loss.item(): 0.0
color_network_dict["point_light_network"].get_light().item(): 5.6220927238464355
1%|▎ | 499/50001 [01:35<3:20:37, 4.11it/s]ic| args.out_dir: './exp_iron_stage2/drv/rabbit'
global_step: 500
loss.item(): 0.014144735410809517
img_loss.item(): 0.0
img_l2_loss.item(): 0.0
img_ssim_loss.item(): 0.0
eik_loss.item(): 0.014144735410809517
roughrange_loss.item(): 0.0
color_network_dict["point_light_network"].get_light().item(): 5.224419593811035

Hi, there must be something wrong on my side.

Could you help me?

Hi, Kai!
I read the code and am confused about the BRDF implemented. According to the code , specular_rgb = light_intensity * specular_albedo * F * D * G / (4.0 * dot + 1e-10) and F = 0.03867, the F is fixed and a 'specular_albedo' is multiplied in BRDF, which is different from simplified Disney BSDF used in PhySG. Could you introduce which BRDF you used?
Thank you!

Awesome work! Is this method only valid for photometric images (captured by a hand-held smartphone camera with LED flashlight turned on in a dark environment)? Can we use images captured by a phone in a common indoor scene?

Hi,
I want to try IRON on my own datasets, but I don't know how to get cam_dict_norm.json, could you share with me this part of code? Thanks!

Hi, I have read this paper and some questions appear. I agree with this paragraph that "For this reason, these methods compute biased gradients w.r.t. the weights of the neural SDF that move surface points along camera rays”. It is easy to understand due to the loss function mentioned in IDR. However, you also display Figure.4 to demonstrate your efficiency which immediately confuses me. Because IDR adopts mask loss to train so that the SDF value will be optimized to fit silhouette supervision and won't lead to similar degradation. Can you give me some inspiration or advice?

. ./train_scene.sh drv/dragon
got Error
FileNotFoundError: [Errno 2] No such file or directory: './data_flashlight/drv/dragon/train/cam_dict_norm.json'

In the Cook-Torrance BRDF
f_specular = D * F * G / (4 * (n dot v) * (n dot l)),
In the co-located setting, you assume v==l,
so the equation should be
f_specular = D * F * G / (4 * (n dot v) ** 2), right?
but in your code implementation,

Hi,
Thank you for this great work!
I tried IRON with my own datasets, when it came to metal materials, the results seems poor, do you know why?

Hi!

Thank you for this great work!

I'm a little confused about export materials. why should repeat uvs to a neighborhood? and is the basic idea to use gaussian as a weight to sum up repeat values?

Finished importing: './exp_iron_stage2/superman/mesh_and_materials_10000/mesh.obj'
Progress: 100.00%

(  0.0002 sec |   0.0000 sec) OBJ Export path: './exp_iron_stage2/superman/mesh_and_materials_10000/mesh.obj'
      ( 93.8459 sec |  93.8253 sec) Finished writing geometry of 'mesh'.
  ( 93.8462 sec |  93.8459 sec) Finished exporting geometry, now exporting materials
  ( 93.8462 sec |  93.8460 sec) OBJ Export Finished

Progress: 100.00%

Error: File format is not supported in file '/IRON/exp_iron_stage2/superman/mesh_and_materials_10000/mesh.obj'

Blender quit
ic| 'Exporting materials...'
Warning: readOBJ() ignored non-comment line 3:

Thank you for your sharing!
I clone your repo and download the dataset. When I run

bash train_scene.sh dataset/luan/buddha

There is an error because we do not have inverse_render.py. Besides, there is an AssertionError.

Hello Wooden
Load data: Begin
Not using masks
image shape, mask shape: torch.Size([80, 768, 1024, 3]) torch.Size([80, 768, 1024, 3])
image pixel range: 0.0 1.0
Load data: End
0%| | 0/100001 [00:00<?, ?it/s]
Traceback (most recent call last):
File "render_volume.py", line 449, in
runner.train()
File "render_volume.py", line 156, in train
self.optimizer.step()
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/site-packages/torch/optim/optimizer.py", line 109, in wrapper
return func(*args, **kwargs)
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/site-packages/torch/optim/adam.py", line 157, in step
adam(params_with_grad,
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/site-packages/torch/optim/adam.py", line 213, in adam
func(params,
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/site-packages/torch/optim/adam.py", line 255, in _single_tensor_adam
assert not step_t.is_cuda, "If capturable=False, state_steps should not be CUDA tensors."
AssertionError: If capturable=False, state_steps should not be CUDA tensors.
ic| args: Namespace(data_dir='./dataset/dataset/luan/buddha/train', eik_weight=0.1, export_all=False, gamma_pred=True, init_light_scale=8.0, inv_gamma_gt=False, is_metal=False, neus_ckpt_fpath='./exp_iron_stage1/dataset/luan/buddha/checkpoints/ckpt_100000.pth', no_edgesample=False, num_iters=50001, out_dir='./exp_iron_stage2/dataset/luan/buddha', patch_size=128, plot_image_name=None, render_all=False, roughrange_weight=0.1, ssim_weight=1.0)
Wrote config file to ./exp_iron_stage2/dataset/luan/buddha/args.txt
Traceback (most recent call last):
File "render_surface.py", line 73, in
shutil.copy2("./inverse_renderer.py", os.path.join(args.out_dir, "code/inverse_renderer.py"))
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/shutil.py", line 435, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/shutil.py", line 264, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: './inverse_renderer.py'
ic| args: Namespace(data_dir='./dataset/dataset/luan/buddha/test', eik_weight=0.1, export_all=False, gamma_pred=True, init_light_scale=8.0, inv_gamma_gt=False, is_metal=False, neus_ckpt_fpath='./exp_iron_stage1/dataset/luan/buddha/checkpoints/ckpt_100000.pth', no_edgesample=False, num_iters=50001, out_dir='./exp_iron_stage2/dataset/luan/buddha', patch_size=128, plot_image_name=None, render_all=True, roughrange_weight=0.1, ssim_weight=1.0)
Wrote config file to ./exp_iron_stage2/dataset/luan/buddha/args.txt
Traceback (most recent call last):
File "render_surface.py", line 73, in
shutil.copy2("./inverse_renderer.py", os.path.join(args.out_dir, "code/inverse_renderer.py"))
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/shutil.py", line 435, in copy2
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/home/hyx/anaconda3/envs/iron/lib/python3.8/shutil.py", line 264, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: './inverse_renderer.py'

Could you please fix these bugs? Thank you for your time!

Even after 100000 steps, it still doesn't get the shape quite right.

Hello!
Thank you for the great work!
Right now I'm trying to reproduce with your code reconstruction of dragon from 200 frames, rendered by mitsuba in 512x512 resolution. Unfortunately after 50000 steps of the second stage I realized that many details on the mesh are lost (if comparing to those pictures from the paper).

Could you please tell me, whether womask_iron.conf config file is the valid one for training with preserving small details? Which parameters might be tuned for better quality? Should be gamma-correction be applied for exr-files generated by mitsuba-renderer before feeding into networks? How input resolution affects shape precision?
Thanks!

Hi，
I read the code and am confused about the BRDF implemented. Why the diffuse_rgb is compute like this:

I get same issue as Issues10:
An error "index 0 is out of bounds for axis 1 with size 0" occurred while exporting the material.
vertices, face_vertices, texturecoords, face_texturecoords = loadmesh_and_checkuv(mesh_fpath, out_dir)
This code gets the shape of the variables texturecoords and face_texturecoords are both (0,0). Why？

I load mesh_fpath using your code:
import igl
vertices, texturecoords, _, face_vertices, face_texturecoords, _ = igl.read_obj('./exp_iron_stage2/xmen/mesh_and_materials_50000/mesh.obj', dtype="float32")
print(vertices.shape, texturecoords.shape, face_vertices.shape, face_texturecoords.shape)

here is the result:
(1432877, 3) (0, 0) (2865856, 3) (0, 0)
Do you know why?

Traceback (most recent call last):
File "render_surface.py", line 549, in
export_mesh_and_materials(export_out_dir, sdf_network, color_network_dict)
File "render_surface.py", line 347, in export_mesh_and_materials
export_materials(os.path.join(export_out_dir, "mesh.obj"), material_predictor, export_out_dir)
File "/home/ecoplants/hdd/changfa/PBR/iron/models/export_materials.py", line 168, in export_materials
vertices, face_vertices, texturecoords, face_texturecoords = loadmesh_and_checkuv(mesh_fpath, out_dir)
File "/home/ecoplants/hdd/changfa/PBR/iron/models/export_materials.py", line 153, in loadmesh_and_checkuv
pcd, pcd_uv = sample_surface(vertices, face_vertices, texturecoords, face_texturecoords, n_samples=10**6)
File "/home/ecoplants/hdd/changfa/PBR/iron/models/export_materials.py", line 51, in sample_surface
A = texturecoords[face_texturecoords[sample_face_idx, 0], :]
IndexError: index 0 is out of bounds for axis 1 with size 0

Hi,
I try to render the synthetic data by running render_rgb_flash_mat.py, the assets are downloaded by bash download_data.sh. But when I run the python file in the terminal of Windows 10 with Mitsuba v3.5.0, there is a error that:

Hi, thanks for you great works! Can you share your data with google drive or Baiduyun?