kakaobrain / flame Goto Github PK

Thanks for your nice work! I downloaded the pretrained models and clip models, and tested them on HumanML3D and Babel datasets. Then, I got a really poor result on both datasets, as shown in following tabels.
I am sure that the models I used for testing were the released models. Have you tested the performance of the released models?

Hi, Thanks for your nice work!
Loss is nan when I set the batchsize as 16 and train the model on 8 RTX 2080Ti via the command python train.py.
How to solve this problem?

Hi, thank you for your brilliant work!

I could not find the smpl_model in the required format at AMASS. The SMPL_python_v.1.1.0 models I found on the AMASS website that are compatible with python have the following format:
smpl/
├── models
│ └── basicmodel_f_lbs_10_207_0_v1.1.0.pkl
│ └── basicmodel_m_lbs_10_207_0_v1.1.0.pkl
│ └── basicmodel_neutral_lbs_10_207_0_v1.1.0.pkl
├── .DS_Store
├── init.py
└── smpl_webuser
└── hello_world
└── hello_smpl.py
└── render_smpl.py
└── lbs.py
└── LICENSE.txt
└── posemapper.py
└── README.txt
└── serialization.py
└── verts.py

Could you please tell me how to get the smpl_model in the format requested below?
smpl_model/
├── female
│ └── model.npz
├── info.txt
├── LICENSE.txt
├── male
│ └── model.npz
└── neutral
└── model.npz

Thank you in advance!

Hi, just noticed that your code for humanml3d process goes like:

NUM_JOINTS = 24
# and 
poses_adjust = motiton_meta["poses"][::fps_adjust_factor]
poses_target = poses_adjust[start_frame:end_frame]
axis_angles = poses_target.reshape(len(poses_target), -1, 3)[
                        :, :NUM_JOINTS, :
                    ] 

This code suggests that you just use the first 24 joint axis angles, but actually, in AMASS, the order of joints is something like:

[22 of body joints]
[left hands indices + right hands indices].
Your order should be something like [:, [0:23]+[37], :] instead of [:, :24,:]

Hi, thanks for your nice work!
When I train the model by running python train.py, the training procedure is getting stuck at Using accelerated ArrayDatatype.
How to solve this problem?

I find code for testing in your repo, but I don't know how to use it

I change my hydra version to 1.1，
Successfully installed antlr4-python3-runtime-4.8 hydra-core-1.1.0 omegaconf-2.1.2

then i run python train.py，the error information is as follows：

[2023-02-17 17:04:46,332][src.training_pipeline][INFO] - Instantiating datamodule <src.datamodules.humanml3d_datamodule.HumanML3DDataModule>
[2023-02-17 17:04:46,415][numexpr.utils][INFO] - Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
[2023-02-17 17:04:46,415][numexpr.utils][INFO] - NumExpr defaulting to 8 threads.
[2023-02-17 17:04:46,584][src.training_pipeline][INFO] - Instantiating model <src.models.diffusion_module.DiffusionLitModule>
Error executing job with overrides: []
Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\osmesa.py", line 22, in GL
return ctypesloader.loadLibrary(
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\ctypesloader.py", line 45, in loadLibrary
return dllType( name, mode )
File "D:\ProgramData\Anaconda3\envs\flame\lib\ctypes_init_.py", line 373, in init
self._handle = _dlopen(self._name, mode)
FileNotFoundError: ("Could not find module 'OSMesa' (or one of its dependencies). Try using the full path with constructor syntax.", 'OSMesa', None)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\utils.py", line 575, in locate
import_module(mod)
File "D:\ProgramData\Anaconda3\envs\flame\lib\importlib_init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in load_unlocked
File "", line 843, in exec_module
File "", line 219, in call_with_frames_removed
File "D:\A-project\flame\flame\src\models\diffusion_module.py", line 29, in
from src.utils.vis_util import render_video_summary
File "D:\A-project\flame\flame\src\utils\vis_util.py", line 14, in
from body_visualizer.mesh.mesh_viewer import MeshViewer
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\body_visualizer\mesh\mesh_viewer.py", line 30, in
import pyrender
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\pyrender_init.py", line 3, in
from .light import Light, PointLight, DirectionalLight, SpotLight
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\pyrender\light.py", line 10, in
from OpenGL.GL import *
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\GL_init.py", line 3, in
from OpenGL import error as _error
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\error.py", line 12, in
from OpenGL import platform, configflags
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform_init.py", line 35, in
load()
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform_init.py", line 32, in _load
plugin.install(globals())
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\baseplatform.py", line 92, in install
namespace[ name ] = getattr(self,name,None)
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\baseplatform.py", line 14, in get
value = self.fget( obj )
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\osmesa.py", line 66, in GetCurrentContext
function = self.OSMesa.OSMesaGetCurrentContext
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\baseplatform.py", line 14, in get
value = self.fget( obj )
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\osmesa.py", line 60, in OSMesa
def OSMesa( self ): return self.GL
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\baseplatform.py", line 14, in get
value = self.fget( obj )
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\OpenGL\platform\osmesa.py", line 28, in GL
raise ImportError("Unable to load OpenGL library", *err.args)
ImportError: ('Unable to load OpenGL library', "Could not find module 'OSMesa' (or one of its dependencies). Try using the full path with constructor syntax.", 'OSMesa', None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "train.py", line 26, in
main()
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra\main.py", line 49, in decorated_main
_run_hydra(
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\utils.py", line 367, in _run_hydra
run_and_report(
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\utils.py", line 214, in run_and_report
raise ex
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\utils.py", line 211, in run_and_report
return func()
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\utils.py", line 368, in
lambda: hydra.run(
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\hydra.py", line 110, in run
_ = ret.return_value
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra\core\utils.py", line 233, in return_value
raise self._return_value
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra\core\utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "train.py", line 22, in main
return train(config)
File "D:\A-project\flame\flame\src\training_pipeline.py", line 48, in train
model: LightningModule = hydra.utils.instantiate(config.model)
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 180, in instantiate
return instantiate_node(config, *args, recursive=recursive, convert=convert)
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 240, in instantiate_node
target = _resolve_target(node.get(_Keys.TARGET))

File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\instantiate_instantiate2.py", line 104, in _resolve_target
return _locate(target)
File "D:\ProgramData\Anaconda3\envs\flame\lib\site-packages\hydra_internal\utils.py", line 577, in _locate
raise ImportError(
ImportError: Encountered error: ('Unable to load OpenGL library', "Could not find module 'OSMesa' (or one of its dependencies). Try using the full path with constructor syntax.", 'OSMesa', None) when loading module 'src.models.diffusion_module.DiffusionLitModule'

Hello.

I tried to reproduce the outputs which is "the person kicks with right leg" with the model name human3dml you uploaded.

And my result shown that the foot contact is unstable. It's not same with your video in front of your github homepage.

So I want to ask you about my problem why it couldn't make the same results you've shown.

Thank you for your kindness answering about issue first.

Found following issues while installing from the scratch:

• Current README.md lacks some required system packages.
• Compatibility broken after releasing the latest pytorch-lightning and transformers.

Hello, thank you for such a great project. I have a question when reading "Table 2: APE and AVE benchmark on the KIT dataset":
The evaluation results of Lin et al. (2018), Language2Pose, Ghosh et al. (2021) and TEMOS are exactly the same as those in the TEMOS paper. However, the comparative experiment in paper is the average of 3 times.

How can this be compared with other papers? Is it fair?
In addition, there is no pre-training model or training and evaluation code for KIT-ML in the project. How can I reproduce your project on KIT-ML?

Hello,

I appreciate your great work on the project.
I have a question regarding the sampling code for generating motion from the text.

The code snippet in 'sample_util.py'
half_eps = uncond_eps + guidance_scale * (cond_eps - uncond_eps)
sample_util.py line 80

Regarding the classifier-free guidance paper, should this line be updated to the following?
half_eps = cond_eps + guidance_scale * (cond_eps - uncond_eps)

I appreciate your time and look forward to your response.

Similar to #5, I still cannot reproduce the results using the released model and the results I got were extremely poor.
{"r_precision": {"top-1": 0.06051829268292683, "top-2": 0.1298780487804878, "top-3": 0.19603658536585367}, "fid": 1481.7516534444785, "clip_score": {"clip_score": 0.14643903637110403}, "mid": -53.25080871582031}

I had installed pytorch-lightning and transformers with correct versions ( 1.8.6 and 4.19.2). I tested the released model on GTX 1080Ti using the command python test.py model=diffusion_hml3d.yaml datamodule=humanml3d.yaml ckpt_path=pretrained/flame_hml3d_bc.ckpt. My python enviorment is shown as follows:

Package                 Version
----------------------- ------------------
absl-py                 1.3.0
aiohttp                 3.8.3
aiosignal               1.3.1
alembic                 1.9.0
antlr4-python3-runtime  4.9.3
asttokens               2.0.5
async-timeout           4.0.2
attrs                   22.1.0
autopage                0.5.1
backcall                0.2.0
black                   22.12.0
body-visualizer         1.1.0
brotlipy                0.7.0
cachetools              5.3.0
certifi                 2022.12.7
cffi                    1.15.0
cfgv                    3.3.1
chardet                 5.1.0
charset-normalizer      2.1.1
click                   8.1.3
cliff                   4.1.0
cmaes                   0.9.0
cmd2                    2.4.2
colorama                0.4.6
colorlog                6.7.0
commonmark              0.9.1
configer                1.4.1
configparser            5.3.0
contourpy               1.0.6
cryptography            37.0.2
cycler                  0.11.0
decorator               5.1.1
distlib                 0.3.6
dotmap                  1.3.30
exceptiongroup          1.0.4
executing               0.8.3
fastjsonschema          2.16.2
filelock                3.9.0
flake8                  6.0.0
fonttools               4.38.0
freetype-py             2.3.0
frozenlist              1.3.3
fsspec                  2022.11.0
ftfy                    6.1.1
fvcore                  0.1.5.post20221221
google-auth             2.15.0
google-auth-oauthlib    0.4.6
greenlet                2.0.1
grpcio                  1.51.1
huggingface-hub         0.11.1
human-body-prior        2.2.2.0
hydra-colorlog          1.2.0
hydra-core              1.3.1
hydra-optuna-sweeper    1.2.0
identify                2.5.11
idna                    3.4
imageio                 2.23.0
imageio-ffmpeg          0.4.7
importlib-metadata      5.2.0
importlib-resources     5.10.1
iniconfig               1.1.1
iopath                  0.1.10
ipython                 8.7.0
isort                   5.11.3
jedi                    0.18.2
jsonschema              4.17.3
jupyter_core            5.1.0
kiwisolver              1.4.4
lightning-utilities     0.4.2
loguru                  0.6.0
Mako                    1.2.4
Markdown                3.4.1
MarkupSafe              2.1.1
matplotlib              3.6.2
matplotlib-inline       0.1.6
mccabe                  0.7.0
mkl-fft                 1.3.1
mkl-random              1.2.2
mkl-service             2.4.0
moviepy                 1.0.3
multidict               6.0.3
mypy-extensions         0.4.3
nbformat                5.7.1
nbstripout              0.6.1
networkx                2.8.8
nodeenv                 1.7.0
numpy                   1.23.4
oauthlib                3.2.2
olefile                 0.46
omegaconf               2.3.0
opencv-python           4.6.0.66
optuna                  2.10.1
packaging               23.0
pandas                  1.5.2
parso                   0.8.3
pathspec                0.10.3
pbr                     5.11.0
pexpect                 4.8.0
pickleshare             0.7.5
Pillow                  9.3.0
pip                     22.3.1
pkgutil_resolve_name    1.3.10
platformdirs            2.6.2
pluggy                  1.0.0
portalocker             2.6.0
pre-commit              2.20.0
prettytable             3.5.0
proglog                 0.1.10
prompt-toolkit          3.0.36
protobuf                3.20.1
psbody-mesh             0.4
psutil                  5.9.4
ptyprocess              0.7.0
pudb                    2022.1.3
pure-eval               0.2.2
pyasn1                  0.4.8
pyasn1-modules          0.2.8
pycodestyle             2.10.0
pycparser               2.21
pyflakes                3.0.1
pygame                  2.1.2
pyglet                  2.0.2.1
Pygments                2.13.0
PyOpenGL                3.1.0
PyOpenGL-accelerate     3.1.5
pyOpenSSL               22.0.0
pyparsing               3.0.9
pyperclip               1.8.2
pyproject_api           1.5.0
pyrender                0.1.45
pyrsistent              0.19.2
PySocks                 1.7.1
pytest                  7.2.0
python-dateutil         2.8.2
python-dotenv           0.21.0
pytorch-lightning       1.8.6
pytorch3d               0.7.2
pytz                    2022.7
PyYAML                  6.0
pyzmq                   24.0.1
regex                   2022.10.31
requests                2.28.1
requests-oauthlib       1.3.1
rich                    12.6.0
rsa                     4.9
scipy                   1.9.3
setuptools              65.5.0
sh                      1.14.3
six                     1.16.0
SQLAlchemy              1.4.45
stack-data              0.2.0
stevedore               4.1.1
tabulate                0.9.0
tensorboard             2.11.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit  1.8.1
tensorboardX            2.5.1
termcolor               2.1.1
tokenizers              0.12.1
toml                    0.10.2
tomli                   2.0.1
torch                   1.12.0
torchaudio              0.12.0
torchmetrics            0.11.0
torchvision             0.13.0
tox                     4.4.5
tqdm                    4.64.1
traitlets               5.8.0
transformers            4.19.2
transforms3d            0.3.1
trimesh                 3.17.1
typing_extensions       4.4.0
urllib3                 1.26.13
urwid                   2.1.2
urwid-readline          0.13
virtualenv              20.17.1
wcwidth                 0.2.5
Werkzeug                2.2.2
wheel                   0.37.1
yacs                    0.1.8
yarl                    1.8.2
zipp                    3.11.0

Hi, I'm trying to train a Motion CLIP using the hyperparameters shown in paper. But I didn't find the introduction about the performance of the trained Motion CLIP in your paper. Could you tell me about the performance of Motion CLIP?