Comments (15)
kevinlin311tw
commented on July 17, 2024
Thanks for the questions.
In our code, we use a unified data format for 2D/3D joints for different datasets.
For 3D joints, the format is [x, y, z, a], where the fourth element indicates the availability of the joint. It is a binary flag. For most cases (such as 3DPW dataset with SMPL style 3D joints), a is always 1. We keep this format for future adaptation to other different 3D joints/mesh topology.
For 2D joints, the format is [x, y, v], where v is the visibility flag.
from meshtransformer.
tqtrunghnvn
commented on July 17, 2024
Thanks so much for your quick reply!
I found that you already reorganized the original 3DPW dataset and managed your own 3DPW dataset, consisting of .tsv files.
The 3DPW dataset originally has train/validation/test split. However, in your customized 3DPW dataset, I only saw train.img.tsv and test.img.tsv.
Could you please clarify this?
from meshtransformer.
tqtrunghnvn
commented on July 17, 2024
Moreover, I also compared the original 3DPW dataset and your customized 3DPW dataset, and I found the difference.
I took the same person and the same frame. The data is much different:
- In the original 3DPW dataset:
joint2d = [[761.0245361328125, 1427.6376953125, 1.0],
[763.7735595703125, 1254.625, 1.0],
[757.8124389648438, 1021.6619262695312, 1.0],
[871.8953247070312, 1010.0325927734375, 1.0],
[838.7183837890625, 1243.81982421875, 1.0],
[812.2440185546875, 1417.615966796875, 1.0],
[709.1945190429688, 1071.94677734375, 1.0],
[708.8104248046875, 942.5911254882812, 1.0],
[724.396728515625, 823.7357177734375, 1.0],
[883.9208984375, 818.1397094726562, 1.0],
[888.2723388671875, 938.8115234375, 1.0],
[897.5230102539062, 1059.4345703125, 1.0],
[807.9381103515625, 771.5740966796875, 1.0],
[802.7290649414062, 636.399658203125, 1.0],
[810.5869140625, 1011.4450073242188, 1.0],
[805.8743896484375, 803.0057373046875, 1.0],
[802.6295166015625, 891.9404907226562, 1.0],
[818.800537109375, 722.0154418945312, 1.0],
[804.0947265625, 668.561279296875, 1.0],
[820.7557983398438, 697.0828857421875, 1.0],
[833.2789916992188, 680.14111328125, 1.0],
[798.1824951171875, 682.2127685546875, 1.0],
[843.4369506835938, 701.8033447265625, 1.0],
[772.8783569335938, 705.4711303710938, 1.0]]
- In your customized 3DPW dataset:
joint2d = [[ 7.67384253e-01 8.47698757e-02 -5.60344048e-01]
[ 8.28211824e-01 3.11791144e-03 -5.87908769e-01]
[ 7.10723322e-01 -9.30041110e-04 -5.61701921e-01]
[ 7.58508424e-01 1.81683114e-01 -5.69663679e-01]
[ 8.45361280e-01 -3.59436410e-01 -5.95123847e-01]
[ 6.94116843e-01 -3.71307514e-01 -5.74784826e-01]
[ 7.57282539e-01 3.13208635e-01 -5.72228007e-01]
[ 8.15533953e-01 -7.35199899e-01 -6.38437082e-01]
[ 7.13418712e-01 -7.46048192e-01 -6.23979217e-01]
[ 7.60892928e-01 3.61799676e-01 -5.52873324e-01]
[ 8.30624588e-01 -7.90706438e-01 -5.26462391e-01]
[ 7.28885027e-01 -7.93635708e-01 -5.09829213e-01]
[ 7.48760967e-01 5.54727070e-01 -6.02829374e-01]
[ 8.22302452e-01 4.70086415e-01 -5.97221059e-01]
[ 6.85288513e-01 4.64970268e-01 -5.87709265e-01]
[ 7.54457942e-01 6.29499406e-01 -5.69330585e-01]
[ 9.07146496e-01 4.77772425e-01 -6.03743600e-01]
[ 5.96054458e-01 4.70962621e-01 -5.83015465e-01]
[ 9.31566392e-01 2.32074144e-01 -6.41171805e-01]
[ 5.78144159e-01 2.32812816e-01 -6.19179812e-01]
[ 9.56104924e-01 6.07977247e-04 -6.08673423e-01]
[ 5.75637751e-01 -4.62294618e-03 -5.72327194e-01]
[ 9.68516244e-01 -7.55253650e-02 -6.00078079e-01]
[ 5.71770439e-01 -8.07533496e-02 -5.57852445e-01]]
These two vectors have not only different dimensions but also very different values. The same situation for joint3d.
Could you please clarify this difference?
Thank you so much, and hope to see your answer soon!
from meshtransformer.
kevinlin311tw
commented on July 17, 2024
We don't use validation set in our experiments. We don't have validation tsv file.
When I look at the first sample in train.label.tsv, I see the expected annotations:
joint2d = [[761.0245361328125, 1427.6376953125, 1.0],
[763.7735595703125, 1254.625, 1.0],
[757.8124389648438, 1021.6619262695312, 1.0],
[871.8953247070312, 1010.0325927734375, 1.0],
[838.7183837890625, 1243.81982421875, 1.0],
[812.2440185546875, 1417.615966796875, 1.0],
[709.1945190429688, 1071.94677734375, 1.0],
[708.8104248046875, 942.5911254882812, 1.0],
[724.396728515625, 823.7357177734375, 1.0],
[883.9208984375, 818.1397094726562, 1.0],
[888.2723388671875, 938.8115234375, 1.0],
[897.5230102539062, 1059.4345703125, 1.0],
[807.9381103515625, 771.5740966796875, 1.0],
[802.7290649414062, 636.399658203125, 1.0],
[810.5869140625, 1011.4450073242188, 1.0],
[805.8743896484375, 803.0057373046875, 1.0],
[802.6295166015625, 891.9404907226562, 1.0],
[818.800537109375, 722.0154418945312, 1.0],
[804.0947265625, 668.561279296875, 1.0],
[820.7557983398438, 697.0828857421875, 1.0],
[833.2789916992188, 680.14111328125, 1.0],
[798.1824951171875, 682.2127685546875, 1.0],
[843.4369506835938, 701.8033447265625, 1.0],
[772.8783569335938, 705.4711303710938, 1.0]]
I couldn't find your provided example in our tsv file.
If you find the vector somewhere in our training pipeline (such as model outputs), it would be more explainable. Please note, following the literature, we did some coordinate normalizations for better training (see https://github.com/microsoft/MeshTransformer/blob/main/metro/datasets/human_mesh_tsv.py#L123).
During inference, we will undo the normalization in order to visualize the joints in the image (see https://github.com/microsoft/MeshTransformer/blob/main/metro/utils/renderer.py#L276).
The operations are commonly used in the literature. We mainly follow GraphCMR for the data pre-processing.
from meshtransformer.
tqtrunghnvn
commented on July 17, 2024
Oh I am sorry for the mistake. The data I got must be:
- In your customized 3DPW dataset:
joint2d = [[761.0245361328125, 1427.6376953125, 1.0],
[763.7735595703125, 1254.625, 1.0],
[757.8124389648438, 1021.6619262695312, 1.0],
[871.8953247070312, 1010.0325927734375, 1.0],
[838.7183837890625, 1243.81982421875, 1.0],
[812.2440185546875, 1417.615966796875, 1.0],
[709.1945190429688, 1071.94677734375, 1.0],
[708.8104248046875, 942.5911254882812, 1.0],
[724.396728515625, 823.7357177734375, 1.0],
[883.9208984375, 818.1397094726562, 1.0],
[888.2723388671875, 938.8115234375, 1.0],
[897.5230102539062, 1059.4345703125, 1.0],
[807.9381103515625, 771.5740966796875, 1.0],
[802.7290649414062, 636.399658203125, 1.0],
[810.5869140625, 1011.4450073242188, 1.0],
[805.8743896484375, 803.0057373046875, 1.0],
[802.6295166015625, 891.9404907226562, 1.0],
[818.800537109375, 722.0154418945312, 1.0],
[804.0947265625, 668.561279296875, 1.0],
[820.7557983398438, 697.0828857421875, 1.0],
[833.2789916992188, 680.14111328125, 1.0],
[798.1824951171875, 682.2127685546875, 1.0],
[843.4369506835938, 701.8033447265625, 1.0],
[772.8783569335938, 705.4711303710938, 1.0]]
- In the original 3DPW dataset:
joint2d = [[8.169990e+02 6.940840e+02 8.994710e-01]
[8.064530e+02 8.138730e+02 9.828750e-01]
[7.282780e+02 8.140330e+02 9.386620e-01]
[7.127130e+02 9.444220e+02 8.730780e-01]
[7.127340e+02 1.064240e+03 9.139100e-01]
[8.845230e+02 8.138130e+02 9.476240e-01]
[8.898530e+02 9.389390e+02 9.635330e-01]
[8.950580e+02 1.048680e+03 8.936900e-01]
[7.648070e+02 1.064310e+03 8.733710e-01]
[7.596690e+02 1.252160e+03 8.256300e-01]
[7.645820e+02 1.424360e+03 7.860840e-01]
[8.532480e+02 1.059310e+03 8.713770e-01]
[8.428340e+02 1.231570e+03 8.600970e-01]
[8.066020e+02 1.408730e+03 8.674500e-01]
[8.064390e+02 6.885450e+02 1.951078e-01]
[8.376640e+02 6.884630e+02 1.843966e-01]
[7.651140e+02 6.991450e+02 1.813974e-01]
[8.479240e+02 6.939170e+02 3.753400e-02]]
from meshtransformer.
tqtrunghnvn
commented on July 17, 2024
By the way, the data for 3D joints is:
- In your customized 3DPW dataset:
joint3d = [[-0.07414655387401581, 0.6259320974349976, 0.23802196979522705, 1.0],
[-0.08293654024600983, 0.2553229033946991, 0.11177857965230942, 1.0],
[-0.11118228733539581, -0.20798233151435852, -0.03652437776327133, 1.0],
[0.11696193367242813, -0.22830316424369812, 0.030360819771885872, 1.0],
[0.07074201852083206, 0.23932282626628876, 0.14836804568767548, 1.0],
[0.03360944241285324, 0.6109831929206848, 0.2613641917705536, 1.0],
[-0.1982978880405426, -0.10928139090538025, 0.02480170875787735, 1.0],
[-0.1994788944721222, -0.3582003712654114, 0.01969926245510578, 1.0],
[-0.17634113132953644, -0.5815433859825134, -0.05405966565012932, 1.0],
[0.13607390224933624, -0.5964640974998474, 0.006664540618658066, 1.0],
[0.1584412306547165, -0.36617332696914673, 0.08553362637758255, 1.0],
[0.18082615733146667, -0.12912505865097046, 0.108730249106884, 1.0],
[-0.017434636130928993, -0.6804583072662354, -0.047672443091869354, 1.0],
[-0.02729572169482708, -0.9356586933135986, -0.0475996695458889, 1.0],
[-0.0037099719047546387, -0.2259705662727356, 0.015443656593561172, 1.0],
[-0.018986718729138374, -0.6225076913833618, -0.03026910126209259, 1.0],
[-0.02061682939529419, -0.4550009071826935, 0.003700997680425644, 1.0],
[-0.005796360783278942, -0.7664101123809814, -0.11061951518058777, 1.0],
[-0.02451862022280693, -0.8751493692398071, -0.04619435593485832, 1.0],
[-0.012012489140033722, -0.8034844398498535, -0.1795082986354828, 1.0],
[0.01684357412159443, -0.8400061726570129, -0.1395934820175171, 1.0],
[-0.04841940104961395, -0.835761547088623, -0.1426447182893753, 1.0],
[0.049128368496894836, -0.8115767240524292, -0.05223245173692703, 1.0],
[-0.08516306430101395, -0.8038434386253357, -0.058558084070682526, 1.0]]
- In the original 3DPW dataset:
joint3d = [[ 7.67384253e-01 8.47698757e-02 -5.60344048e-01]
[ 8.28211824e-01 3.11791144e-03 -5.87908769e-01]
[ 7.10723322e-01 -9.30041110e-04 -5.61701921e-01]
[ 7.58508424e-01 1.81683114e-01 -5.69663679e-01]
[ 8.45361280e-01 -3.59436410e-01 -5.95123847e-01]
[ 6.94116843e-01 -3.71307514e-01 -5.74784826e-01]
[ 7.57282539e-01 3.13208635e-01 -5.72228007e-01]
[ 8.15533953e-01 -7.35199899e-01 -6.38437082e-01]
[ 7.13418712e-01 -7.46048192e-01 -6.23979217e-01]
[ 7.60892928e-01 3.61799676e-01 -5.52873324e-01]
[ 8.30624588e-01 -7.90706438e-01 -5.26462391e-01]
[ 7.28885027e-01 -7.93635708e-01 -5.09829213e-01]
[ 7.48760967e-01 5.54727070e-01 -6.02829374e-01]
[ 8.22302452e-01 4.70086415e-01 -5.97221059e-01]
[ 6.85288513e-01 4.64970268e-01 -5.87709265e-01]
[ 7.54457942e-01 6.29499406e-01 -5.69330585e-01]
[ 9.07146496e-01 4.77772425e-01 -6.03743600e-01]
[ 5.96054458e-01 4.70962621e-01 -5.83015465e-01]
[ 9.31566392e-01 2.32074144e-01 -6.41171805e-01]
[ 5.78144159e-01 2.32812816e-01 -6.19179812e-01]
[ 9.56104924e-01 6.07977247e-04 -6.08673423e-01]
[ 5.75637751e-01 -4.62294618e-03 -5.72327194e-01]
[ 9.68516244e-01 -7.55253650e-02 -6.00078079e-01]
[ 5.71770439e-01 -8.07533496e-02 -5.57852445e-01]]
Could you please elaborate on this difference and share with us your code for creating 3DPW tsv files?
from meshtransformer.
kevinlin311tw
commented on July 17, 2024
I am not very sure about the format as you pointed out here. I think there are three possible reasons for this problem.
- The joint order may be different.
- We may use different normalizations (with image size and/or root coordinate).
- We may use different coordinate systems.
As I can recall, I actually tried to normalize all these factors before. It has been a while, but I remember that our 3D joints will look similar to the original 3DPW ones as you post here.
Unfortunately, the tsv generation code is missing. The code was written 1 year ago, and I can't find it now. I need some time to dig it out. I may need to rewrite a new code when I have time. I will add it to the repo then.
Thanks a lot for your interests in our work :)
from meshtransformer.
tqtrunghnvn
commented on July 17, 2024
You mean you normalized both joint_2d and joint_3d before writing them into tsv file.
If that is right, why do you still normalize joint_2d and joint_3d in your dataloader code?
from meshtransformer.
kevinlin311tw
commented on July 17, 2024
We did normalization in the dataloader. Please stay tuned for the tsv generation code!
from meshtransformer.
tqtrunghnvn
commented on July 17, 2024
As far as I figure out, you only provided the tsv generation code for pseudo-3d-dataset (https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo.py)
There is no tsv code for 3DPW dataset. If you can provide it soon, it would be highly appreciated!
Thank you so much!
from meshtransformer.
kevinlin311tw
commented on July 17, 2024
The 3dpw tsv example code can be found at https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_3dpw.py
from meshtransformer.
wjingdan
commented on July 17, 2024
The 3dpw tsv example code can be found at https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_3dpw.py
Can you provide the code to generate the TSV file of the human36m dataset? Thank you very much.
from meshtransformer.
Hao-hui
commented on July 17, 2024
The 3dpw tsv example code can be found at https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_3dpw.py
Can you provide the code to generate the TSV file of the human36m dataset? Thank you very much.
Hi, wjingdan
I am interested in the authors' code to generate the TSV file of the human36m dataset. Or do you know the difference between TSV file of human36m and the original human36m.
Thank you!
from meshtransformer.
kevinlin311tw
commented on July 17, 2024
Hi all,
You may find the example script to reproduce Human3.6M tsv files below:
https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_h36m.py
from meshtransformer.
WZS666
commented on July 17, 2024
Hello @kevinlin311tw ,
Have you ever get Out of memory (killed) and end the process while writing things in to tsv?
I'm trying to build my own tsv files, but trouble with OOM (killed) problem.
If you have any suggestion. Please let's me know.
Thanks a lot !
from meshtransformer.
Related Issues (20)
- This repo is missing important files
- How to visualize attention correlation as the paper
- Training on single dataset HOT 1
- Output Score / Confidence
- Project dependencies may have API risk issues
- Cannot render my hand by my photo HOT 1
- About the template vertices and joints
- Question about training with hand dataset? HOT 1
- Can't download 'https://datarelease.blob.core.windows.net/metro/datasets/filename.tar'
- pre-trained models！ERROR 400: Bad Request.
- Demo Lack of three-digit reconstruction effect，缺少三位重建 HOT 1
- COCO SMPL Data Missing
- About DDP train on the specified gpu
- It is stopped in 2023-10-31 16:00:44,169 METRO Inference INFO: Using 1 GPUs
- Exception: Unable to get url: http://files.is.tue.mpg.de/mloper/opendr/osmesa/OSMesa.Linux.aarch64.zip HOT 1
- Question about the learning rate adjustment strategy
- Question about weight decay setting
- Pre-trained model licensing
- How to do single-player multi-card training？ HOT 1
- Apex version incompatible HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from meshtransformer.