jackzhousz / dreamfields-3d

A colab friendly toolkit to generate 3D mesh model / video / nerf instance / multiview images of colourful 3D objects by text and image prompts input, based on dreamfields.

dreamfields-3D

A colab friendly toolkit to generate 3D mesh model / video / NeRF instance / multiview images of colourful 3D objects by text and image prompts input. Edited by Shengyu Meng (Simon)

Check the colab notebook for usage.

Dreamfields-3D is modified from dreamfields-torch and dreamfields, please check the Credits.md for details.

Update logs

Beta v0.6 : Apply random fovy (view angle) in training; update image augmentation before feed into CLIP; improve training stability and performance. (2022-09-25)

ngp_200_rgb-.mp4

Example generated by text prompt: "a cyborg organic biological pavilion could breathe with building skin containing algae, in the style of dezeen, trending on artstation, surreal", with CLIP ViT-L/14 model, training for 200 epochs with clip_aug and random_fovy_training mode enabled.

Preview.mp4

Example generated by text prompt: "a beautiful painting of a flower tree, by Chiho Aoshima, Long shot, surreal", with CLIP ViT-L/14 model, training for 200 epochs.

Main Contributions:

• Export obj & ply model with vertex colour.
• Export 360° Video of final model.
• Visualizing the training progress and preview the output video in colab.
• Improve the generation quality.
  • Allow to use different CLIP models.
  • Improve the pre-process of the renderings before feeding into CLIP.
  • Apply random view angle in training.
• Add more useful augments.
• Organize the colab notebook.

Future update plan:

• Use different CLIP models simultaneously.
• Apply the image prompt only in certain assigned direction.
• Convert existing mesh to NeRF instance then modify by text / image prompts.
• Reduce GPU RAM occupation in training.

Becuase I am a coding beginner, cannot promise for the progress. Welcome to contact me to contribute to this repository.

Compatibility:

• About system:
  • Colab: Pass on google Colab (tested on A100/v100/P100 GPU at 08/09/2022, GPU less than 16G RAM need to lower the clip_w_h and clip_aug_copy volume)
  • Ubuntu: The previous version (dreamfields-torch) has successfully ran on Ubuntu 18.04 with RTX 3090. No test has been done for the dreamfields-3D yet, but it should be fined.
  • Windows: It should work in windows with proper environment, but I failed to build the raymarching extension in different windows machine. More test will be required.
  • For locally running, please refer to the following installation guide from dreamfields-torch.
• About Exporting 3D model:
  • The 3D models will be exported as obj and ply format with vertex colour.
  • Mesh with vertex colour could be directly viewed in meshLab and Rhino3D.
  • For viewing colour in Blender, please import the ply format model first (obj supported in latest 3.3 version), then create a new material, and plug a Color Attribute node into the base color in shader editor, then you should see the vertex colour.
• About GUI:
  • GUI is supported only when running locally. However, some new features maybe not available in GUI model yet. I would recommend to train without GUI first, then run test on the pretrained model in GUI, to browse the final 3D NeRF instance.

👇 Bellow readme was modified from the dreamfields-torch repository.

---

dreamfields-torch (WIP)

A pytorch implementation of dreamfields as described in Zero-Shot Text-Guided Object Generation with Dream Fields.

An example of a generated neural field by prompt "cthulhu" viewed in real-time:

cthulhu.mp4

Install

The code framework is based on torch-ngp.

git clone https://github.com/shengyu-meng/dreamfields-3D.git
cd dreamfields-3D

Install with pip

pip install -r requirements.txt

install customized verion of pymarchingcubes

bash scripts/install_PyMarchingCubes.sh

Build extension

# install all extension modules
bash scripts/install_ext.sh
# if you want to install manually, here is an example:
cd raymarching
python setup.py build_ext --inplace # build ext only, do not install (only can be used in the parent directory)
pip install . # install to python path (you still need the raymarching/ folder, since this only install the built extension.)

Tested environments

(For dreamfields-torch, not for dreamfileds-3D)

• Ubuntu 20 with torch 1.10 & CUDA 11.3 on a TITAN RTX.
• Windows 10 with torch 1.11 & CUDA 11.3 on a RTX 3070.

Currently, --ff only supports GPUs with CUDA architecture >= 70. For GPUs with lower architecture, --tcnn can still be used, but the speed will be slower compared to more recent GPUs.

Usage

First time running will take some time to compile the CUDA extensions.

# text-guided generation
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16

# use the GUI
python main_nerf.py --text "cthulhu" --workspace trial --cuda_ray --fp16 --gui

# [experimental] image-guided generation (also use the CLIP loss)
python main_nerf.py --image /path/to/image --workspace trial --cuda_ray --fp16

check the scripts directory for more provided examples.

Difference from the original implementation

• Mip-nerf is not implemented, currently only the original nerf is supported.
• Sampling poses with an elevation range in [-30, 30] degrees, instead of fixed at 30 degree.
• Use the origin loss.

Update Logs

• 5.18: major update.
• 3.16: basic reproduction.

Acknowledgement

• The great paper and official JAX implementation of dreamfields:

  @article{jain2021dreamfields,
      author = {Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben},
      title = {Zero-Shot Text-Guided Object Generation with Dream Fields},
      journal = {arXiv},
      month = {December},
      year = {2021},
  }   
  

• The GUI is developed with DearPyGui.