wangxihao / icon

ICON: Implicit Clothed humans Obtained from Normals

ICON: Implicit Clothed humans Obtained from Normals

arXiv, December 2021.
Yuliang Xiu · Jinlong Yang · Dimitrios Tzionas · Michael J. Black

ICON.mp4

Table of Contents

1. Who needs ICON
2. TODO
3. Installation
4. Demo
5. Citation
6. Acknowledgments
7. Disclosure
8. License
9. Contact

Who needs ICON?

• If you want to reconstruct 3D clothed humans in unconstrained poses from in-the-wild images
  • together with the body under clothing (e.g. SMPL, SMPL-X)
  • clothed-body normal maps (front/back) predicted from images

ICON's outputs from single RGB image

• If you want to obtain a realistic and animatable 3D clothed avatar direclty from video / a sequence of monocular images
  • fully-textured with per-vertex color
  • could be animated by SMPL pose parameters
  • with pose-dependent clothing deformation

3D Clothed Avatar, created from 400+ images using ICON+SCANimate, animated by AIST++

TODO

• Testing code and pretrained model(*self-implemented version)
  • ICON (w/ & w/o global encoder)
  • PIFu* (RGB image + predicted normal map as input)
  • PaMIR* (RGB image + predicted normal map as input)
• Online app
• Training code
• Dataset processing code
• Video2Avatar module

Installation

Please follow the Installation Instruction to setup all the required packages, extra data, and models.

Demo

cd ICON/apps

# PIFu* (*: re-implementation)
python infer.py -cfg ../configs/pifu.yaml -gpu 0 -in_dir ../examples -out_dir ../results

# PaMIR* (*: re-implementation)
python infer.py -cfg ../configs/pamir.yaml -gpu 0 -in_dir ../examples -out_dir ../results

# ICON w/ global filter (better visual details --> lower Normal Error))
python infer.py -cfg ../configs/icon-filter.yaml -gpu 0 -in_dir ../examples -out_dir ../results

# ICON w/o global filter (higher evaluation scores --> lower P2S/Chamfer Error))
python infer.py -cfg ../configs/icon-nofilter.yaml -gpu 0 -in_dir ../examples -out_dir ../results

More Qualitative Results

Comparison with other state-of-the-art methods
Reconstruction on in-the-wild photos with extreme poses (GIF)
Reconstruction on in-the-wild photos with extreme poses (PNG)
Predicted normals on in-the-wild images with extreme poses

Citation

@article{xiu2021icon,
  title={ICON: Implicit Clothed humans Obtained from Normals},
  author={Xiu, Yuliang and Yang, Jinlong and Tzionas, Dimitrios and Black, Michael J},
  journal={arXiv preprint arXiv:2112.09127},
  year={2021}
}

Acknowledgments

We thank Yao Feng, Soubhik Sanyal, Qianli Ma, Xu Chen, Hongwei Yi, Chun-Hao Paul Huang, and Weiyang Liu for their feedback and discussions, Tsvetelina Alexiadis for her help with the AMT perceptual study, Taylor McConnell for her voice over, Benjamin Pellkofer for webpage, and Yuanlu Xu's help in comparing with ARCH and ARCH++.

Special thanks to Vassilis Choutas for sharing the code of bvh-distance-queries

Here are some great resources we benefit from:

• MonoPortDataset for Data Processing
• PaMIR, PIFu, PIFuHD, and MonoPort for Benchmark
• SCANimate and AIST++ for Animation
• rembg for Human Segmentation
• smplx, PARE, PyMAF, and PIXIE for Human Pose & Shape Estimation
• CAPE and THuman for Dataset
• PyTorch3D for Differential Rendering

Some images used in the qualitative examples come from pinterest.com.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No.860768 (CLIPE Project).

Disclosure

MJB has received research gift funds from Adobe, Intel, Nvidia, Facebook, and Amazon. While MJB is a part-time employee of Amazon, his research was performed solely at, and funded solely by, Max Planck. MJB has financial interests in Amazon, Datagen Technologies, and Meshcapade GmbH.

License

This code and model are available for non-commercial scientific research purposes as defined in the LICENSE file. By downloading and using the code and model you agree to the terms in the LICENSE.

Contact

For more questions, please contact [email protected]

For commercial licensing, please contact [email protected]