thu-lyj-lab / t3bench

T3Bench: Benchmarking Current Progress in Text-to-3D Generation

📃 Paper • 🌐 Project Page

T³Bench: Benchmarking Current Progress in Text-to-3D Generation

T³Bench is the first comprehensive text-to-3D benchmark containing diverse text prompts of three increasing complexity levels that are specially designed for 3D generation (300 prompts in total). To assess both the subjective quality and the text alignment, we propose two automatic metrics based on multi-view images produced by the 3D contents. The quality metric combines multi-view text-image scores and regional convolution to detect quality and view inconsistency. The alignment metric uses multi-view captioning and Large Language Model (LLM) evaluation to measure text-3D consistency. Both metrics closely correlate with different dimensions of human judgments, providing a paradigm for efficiently evaluating text-to-3D models.

🔥 Updates

[2023/10/24] We have released mesh results of all prompt sets and methods! Please check here to download.

Evaluate on T³Bench

Environment Setup

We adopt the implementation of ThreeStudio to test the current text-to-3D methods. Please first follow the instructions of ThreeStudio to setup the generation environment.

Then install the following packages used for evaluation:

pip install -r requirements.txt

Note that we use a slightly modified version of ThreeStudio to ensure efficient generation.

Evaluation

Run Text-to-3D and Extract Mesh

# YOUR_GROUP: Choose the prompt set to test, including [single, surr, multi]
# YOUR_METHOD: We now support latentnerf, magic3d, fantasia3d, dreamfusion, sjc, and prolificdreamer.
python run_t3.py --group YOUR_GROUP --gpu YOUR_GPU --method YOUR_METHOD
python run_mesh.py --group YOUR_GROUP --gpu YOUR_GPU --method YOUR_METHOD

Quality Evaluation

python run_eval_quality.py --group YOUR_GROUP --gpu YOUR_GPU --method YOUR_METHOD

Alignment Evaluation

# First get the 3D prompt of the text-to-3D result
python run_caption.py --group YOUR_GROUP --gpu YOUR_GPU --method YOUR_METHOD
# then run the LLM Evaluation
python run_eval_alignment.py --group YOUR_GROUP --gpu YOUR_GPU --method YOUR_METHOD

Citation

@misc{he2023t3bench,
      title={T$^3$Bench: Benchmarking Current Progress in Text-to-3D Generation}, 
      author={Yuze He and Yushi Bai and Matthieu Lin and Wang Zhao and Yubin Hu and Jenny Sheng and Ran Yi and Juanzi Li and Yong-Jin Liu},
      year={2023},
      eprint={2310.02977},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Acknowledgement

This project could not be possible without the open-source works from ThreeStudio, Cap3D, Stable-DreamFusion, ImageReward, LAVIS. We sincerely thank them all.

t3bench's Issues

Question regarding the alignment scores

If I get an average alignment score 3,
then the normalized score is 3/5x100 = 60
or (3-1)/4x100 = 50 ?
Currently based on your paper sec 4.2, I assume the second calculation is correct.
Is that right?

Submit results on t3bench

Hello, thank you for providing a comprehensive benchmark for 3D generation. I am the author of GaussianDreamer and I would like to know how to submit our results on T3bench. Here are our results.

Single Obj.	** Single w/ Surr.**	Multi Obj.	Average
54.0	48.6	34.5	45.7

Question regarding text prompts

Hello! Thanks for your interesting work. I have a question regarding the text prompts you release in this benchmark.
I am wondering if you are providing triplets of the increasingly descriptive text prompts for every object. or if every text prompt is referring to a random object.

For example:

• single object : "A red fire hydrant"
• single object with surroundings: "A red fire hydrant on a sidewalk"
• multiple objects: "A red fire hydrant is on a sidewalk close to a large tree".

Release human annotation?

Would it be possible to also release the human annotations? Thanks!

Generation time

Hello, this is a wonderful job. I would like to know the generation time of these methods (ProlificDreamer, SJC, Fantasia3D, LatentNeRF, Magic3D, Dreamfusion) in the paper?

There's a conflict in requirements.txt

When using pip install for requirement.txt, salesforce-lavis, and image-reward are in conflict when sharing the transformer. They have different requirements for the version of the transformer.