Comments (7)
So yea, what I am doing now is like this:
for a basic 3D object we need geometry + texture.
from the wonder3D I get the geometry, it's good enough for my needs.
Then I use the second stage of dreamgaussian repositry/paper, they have a differentiable rendered for texture refinement.
you need to make sure the 3D obj from wonder3d aligned with the renderer.
so I optimize the Object texture using my multi-view images and that renderer.
from wonder3d.
Hello, thanks for your interests in our work.
Currently our diffusion model only support 256x256 resolution, so 512 does not work.
For the OOM problem, you should use 1gpu.yaml, the diffusion won't cost too much memory. Current inference code supports float32, I will change it to float16 later to save memory.
For stage 2, I will write a document about the parameters to favor better reconstruction methods.
from wonder3d.
First of all, amazing work on this project!
Would it be possible to increase texture quality? I've tried increasing the img_wh under validation_dataset in the config to [512, 512] but I get:
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 32 but got size 64 for tensor number 1 in the list.
As for second phase, what relevant parameters in the 'neuralangelo-ortho-wmask.yaml' config would allow modification to mesh quality?
And finally, less relevant to the above, but I've tried running on multi-GPU and the seconds phase (instant-nsr-pl) works fine when reducing the iterations but when trying to run the diffusion phase with a similar config to 8gpu.yaml but with 4 instead (running on 4xNVIDIA L4) I get OOM. Any tips you can share on this?
Thanks a lot!
Hello! I met this problem, too.
I find that img_wh
is corresponding to the size of latents between unet and vae. So changing the sample_size
of unet in mvdiffusion-joint-ortho-6views.yaml into 64 will make their size the same and fix the bug. And setting crop_size
to 384 leads to a better image proportion. But I can only get inconsistent results, not as amazing as the result of 256x256.
Hope authors can provide a higher-resolution pretrained unet or even training code!
from wonder3d.
Hello, @flamehaze1115 that would very appreciated if you could share a reconstruction doc !!
@jclarkk, I will try to answer with what I did, I still experiment it so maybe in the future would be better:
- The reconstruction stage starts first of all with resizeing the 256X256 images to 1024X1024. Now First Make sure when resizing the resolution wouldn't decrease a lot. Second make sure the object is scaled/aligned exactly to 1024 (and not small in the middle with lot of whot margins) . Third: you can manually change the front image with your input image (make sure it's aligned with the front_normals image, you can make sure of this by crop=-1 in the diffusion first stage).
- If you resize it to 2048X2048 it would be better (of course take more time to reconstruction and more memory)
- In the config there is a weights for each view, give the front view the highest weight.
- (I am not sure of this) you can do manual postprocess using blender or any other software, to map the texture+normals to the front view (this will make sure, depending in the geometry, that your texture will be good)
- (I will try in the future) play with the optimizer parameters.
- (I will try it in the future and I am not sure if I understand it right (appreciate feedback if you know the theory)) because they use NeUs, they embed the points colors to mlp network ( like nerf? ) and this would give little bit blurry for gentle small details if there is no enough input data (just 6) ), so by removing this mlp and map it as it is, the color would be nice (that wouldn't work for reflections and mirrors...)
from wonder3d.
@fefespn
Thanks a lot for your notes, I'll try to play around with this as well.
from wonder3d.
@fefespn Hi, so have you tried with your suggestions? And, do you have any further good suggestions? Thanks
from wonder3d.
So yea, what I am doing now is like this: for a basic 3D object we need geometry + texture. from the wonder3D I get the geometry, it's good enough for my needs. Then I use the second stage of dreamgaussian repositry/paper, they have a differentiable rendered for texture refinement. you need to make sure the 3D obj from wonder3d aligned with the renderer. so I optimize the Object texture using my multi-view images and that renderer.
@fefespn Hello! Can you share more about your modified second stage of dg? The different camera settings of these two methods are a little confusing.
from wonder3d.
Related Issues (20)
- When does the background turn white ? HOT 2
- Updated ubuntu library - Failed to fetch old library
- About camera_embedding at inference
- Changing Poses for 3D Reconstruction
- Fine-tuning Problem HOT 2
- vvbmbmn
- 在gradio图像界面上生成六视角图片时出错
- where can I get obj files in lvis_uids_filter_by_vertex.json
- Is there any way to make the model faster?
- Images all black
- Stage 2 training BUG: following keys are missing HOT 3
- Operator bpy.ops.transform.rotate.poll() failed, context is incorrect
- Bad texture using default settings for owl.png
- Can I use a pinhole image for the inference input?
- Instant-nsr-pl为什么运行一直卡着? HOT 1
- 我使用NeuS进行mesh extraction,效果这样是合理的吗?
- About the cleanup process on the LVIS subset of the Obja verse dataset
- 有没有类似中文教程?权重安装太慢?Are there similar Chinese courses? Weight installation Too Slow? HOT 5
- inconsistent noise for the two domain of different view of the same object?
- How to render a object with roughness/metallic textures and output roughness/metallic images?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from wonder3d.