junxinghu / choi

CHOI

This repository is the official implementation of the following paper:

Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images

Junxing Hu, Hongwen Zhang, Zerui Chen, Mengcheng Li, Yunlong Wang, Yebin Liu, Zhenan Sun

AAAI, 2024

[Project Page] [Paper]

Requirements

• Python 3.8

conda create --no-default-packages -n choi python=3.8
conda activate choi

• PyTorch is tested on version 1.8.0

conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1.1 -c pytorch -c conda-forge

• Other packages are listed in requirements.txt

pip install -r requirements.txt

Pre-trained Model and Dataset

• Unzip weights.zip and the pre-trained model is placed in the ./weights/ho3d/checkpoints directory

• Unzip data.zip and the processed data and corresponding SDF files are placed in the ./data directory

• Download the HO3D dataset and put it into the ./data/ho3d directory

• Download the MANO model MANO_RIGHT.pkl and put it into the ./externals/mano directory

Evaluation

• To evaluate my model on HO3D, run:

python -m models.choi --config-file experiments/ho3d.yaml --ckpt weights/ho3d/checkpoints/ho3d_weight.ckpt

• The resulting file is generated in the ./output directory

Results

Our method achieves the following performance on the HO3D test set:

Method	F@5mm	F@10mm	Chamfer Distance (mm)
CHOI (Ours)	0.393	0.633	0.646

For video inputs from the OakInk dataset:

^{The video is reconstructed frame-by-frame without post-processing. The objects are unseen during the training.}

More results: Project Page

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{hu2024learning,
  title={Learning Explicit Contact for Implicit Reconstruction of Hand-held Objects from Monocular Images},
  author={Hu, Junxing and Zhang, Hongwen and Chen, Zerui and Li, Mengcheng and Wang, Yunlong and Liu, Yebin and Sun, Zhenan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  year={2024}
}

Acknowledgments

Part of the code is borrowed from IHOI, Neural Body, and MeshGraphormer. Many thanks for their contributions.

choi's Issues

weights.zip and data.zip on Google Drive

First of all, congratulations on your acceptance to AAAI 2024!

For people who cannot access Baidu Drive, may I ask for uploading the files to Google Drive or OneDrive?

Checkpoint for model trained on OakInk

May I ask for the checkpoint of the CHOI model trained on OakInk?

Errors of weights.zip and data.zip

Hi, @JunxingHu. It is a great job for reconstructing hand-object interaction. When I reproduce the results, the weights.zip and data.zip cannot be decompressed. Could you reupload these two files or provide the download link.

Looking forward to your reply.