3d-hand-shape / hand-graph-cnn Goto Github PK
View Code? Open in Web Editor NEW3D Hand Shape and Pose Estimation from a Single RGB Image
License: Other
3D Hand Shape and Pose Estimation from a Single RGB Image
License: Other
Hi @3d-hand-shape ,
First of all , thanks for your great work!
However, I've got a problem with using your model, i managed to change the code to test my own data, I tried with hand detector and without hand detector , result is really bad. here I am sharing two output, the result is in your data is good, but in other images is bad
Please help,
Hi @raingo ,
Thanks for your great paper,
Thank you so much in advance
As it is mentioned in
STB
readme, we have GT coordinate inworld coordinate system
,right?, and providedrotation & translation vector
is to transfer coordinate from depth camera to color camera, right? Why you use them for transforming from world coordinate??
would you please give me a reference for writhing
SK_rot_mx
function which should beRodrigues' rotation formula
?
second when you transfer back from network estimation to camera coordinate , why have you used just depth of root , not all coordinate of root, (
uvd2xyz function in image_uti.py
) ?
Hello, I would like to make my own synthectic dataset, is there any tools or tutorial to make it?
May I know what is Snap Joint'' and why do we need to convert the STB index to Snap index? Is
Snap joint'' given by the synthetic dataset collected by your team?
Thank you.
Hi,Author
I was complete test my own image,and i have a question that how to write the hand mesh to obj file according to the (8, 954, 3) vertices? how to generate vt and f? Can you help me,Thanks.
I can't get the pre-train model by this link,https://mega.nz/#!yfZXBayC!izaLXi4X8LsgPuRWqKlUrCKBWNLVKTvfgAuFIS7SSFY.I don;t kown how to deal with it.I will look for ward your reply and help me .Thank you.
Thanks for your great work!
I notice that you have given the real-world meshes, and the paper didn't make it clear how you get this. Could you please direct me into this?
Thanks in advance for your response~!
When running the script on real world data as per example I get the following error.
2019-07-16 16:13:25,003 hand_shape_pose_inference INFO: Evaluate: [8/583] Average pose estimation error: 10.21 (mm)
2019-07-16 16:13:25,003 hand_shape_pose_inference INFO: Saving image: ./output\configs/eval_real_world_testset.yaml\pred_0.jpg
Traceback (most recent call last):
File "/hand-graph-cnn/eval_script.py", line 111, in <module>
main()
File "/hand-graph-cnn/eval_script.py", line 99, in main
est_pose_cam_xyz, file_name)
File "\hand-graph-cnn\hand_shape_pose\util\vis.py", line 176, in save_batch_image_with_mesh_joints
rend_img_overlay, rend_img_vp1, rend_img_vp2 = draw_mesh(mesh_renderer, image, cam_param, box, mesh_xyz)
File "\hand-graph-cnn\hand_shape_pose\util\vis.py", line 55, in draw_mesh
rend_img_overlay = mesh_renderer(mesh_xyz, cam=cam_for_render, img=image, do_alpha=True)
File "\hand-graph-cnn\hand_shape_pose\util\renderer.py", line 79, in __call__
color_id=color_id)
File "\hand-graph-cnn\hand_shape_pose\util\renderer.py", line 225, in render_model
imtmp = simple_renderer(rn, verts, faces, color=color)
File "\hand-graph-cnn\hand_shape_pose\util\renderer.py", line 179, in simple_renderer
return rn.r
File "\lib\chumpy\ch.py", line 555, in r
self._call_on_changed()
File "\lib\chumpy\ch.py", line 550, in _call_on_changed
self.on_changed(self._dirty_vars)
File "\lib\opendr\renderer.py", line 1080, in on_changed
self.vbo_verts_face.set_array(np.array(self.verts_by_face).astype(np.float32))
AttributeError: 'ColoredRenderer' object has no attribute 'vbo_verts_face'
Process finished with exit code 1
I found online a solution for this problem described here: polmorenoc/opendr#1
and added in util\renderer.py the following lines just before "return rn.r" in def simple_renderer(...)
flipXRotation = np.array([[1.0, 0.0, 0.0, 0.0],
[0.0, -1.0, 0., 0.0],
[0.0, 0., -1.0, 0.0],
[0.0, 0.0, 0.0, 1.0]])
rn.camera.openglMat = flipXRotation # this is from setupcamera in utils
rn.glMode = 'glfw'
rn.sharedWin = None
rn.overdraw = True
rn.nsamples = 8
rn.msaa = True # Without anti-aliasing optimization often does not work.
rn.initGL()
now I get a crash in "draw_color_image()" in opendr\renderer saying I am missing "visibillity_image" variable.
This is where I got stuck, can you please advise?
thank you
For the full model you used to train the MLP network, is that the model fine-tuned with STB dataset or the model just trained with synthetic dataset?
Thanks in advance.
Where is the training code ??
Hello ~
Nice jobs!
Could you share me with the data for training?
params.mat & pose_gt.mat for my own dataset to create the outputs...! @3d-hand-shape @geliuhao @raingo
RuntimeError: Error(s) in loading state_dict for Net_HM_HG:
Missing key(s) in state_dict: "bn1.num_batches_tracked", "r1.bn.num_batches_tracked", "r1.bn1.num_batches_tracked", "r1.bn2.num_batches_tracked", "r4.bn.num_batches_tracked", "r4.bn1.num_batches_tracked", "r4.bn2.num_batches_tracked", "r5.bn.num_batches_tracked", "r5.bn1.num_batches_tracked", "r5.bn2.num_batches_tracked", "hourglass.0.low2.low2.low2.low2_.0.bn.num_batches_tracked", "hourglass.0.low2.low2.low2.low2_.0.bn1.num_batches_tracked", "hourglass.0.low2.low2.low2.low2_.0.bn2.num_batches_tracked", "hourglass.0.low2.low2.low2.low2_.1.bn.num_batches_tracked", "hourglass.0.low2.low2.low2.low2_.1.bn1.num_batches_tracked"
How could I fix it?
I got this error, and I think it's because of different python version.
I implemented the inference code, eval_script.py, with python 2.
Traceback (most recent call last):
File "eval_script.py", line 111, in <module>
main()
File "eval_script.py", line 52, in main
model = ShapePoseNetwork(cfg, output_dir)
File "/home/ubuntu/3d_pose_estimation/hand-graph-cnn/hand_shape_pose/model/shape_pose_network.py", line 41, in __init__
build_hand_graph(cfg.GRAPH.TEMPLATE_PATH, output_dir)
File "/home/ubuntu/3d_pose_estimation/hand-graph-cnn/hand_shape_pose/util/graph_util.py", line 154, in build_hand_graph
graph_dict = np.load(graph_dict_path, allow_pickle=True).item()
File "/home/ubuntu/anaconda3/envs/py27/lib/python2.7/site-packages/numpy/lib/npyio.py", line 447, in load
pickle_kwargs=pickle_kwargs)
File "/home/ubuntu/anaconda3/envs/py27/lib/python2.7/site-packages/numpy/lib/format.py", line 701, in read_array
array = pickle.load(fp, **pickle_kwargs)
ValueError: unsupported pickle protocol: 3
How can I fix it?
I'll be looking forward to your reply.
Hi @3d-hand-shape
Thanks in advance for your helpful comments
Have you had data augmentation (random rotation translation of image and 3D labels) during training, as I found in data loader provided for evaluation not any specific augmentation, is this data loader used for training too?
Could you provide the code about PCK?
I am looking forward to your reply.
I followed the instructions and I ran the first line of code on my CMD :- python eval_script.py --config-file "configs/eval_real_world_testset.yaml" and I got the following error.
Traceback (most recent call last):
File "eval_script.py", line 111, in <module>
main()
File "eval_script.py", line 78, in main
model(images, cam_params, bboxes, pose_roots, pose_scales)
File "C:\Users\ROHIT SANJAY\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "C:\Users\ROHIT SANJAY\Desktop\hand-graph-cnn-master\hand_shape_pose\model\shape_pose_network.py", line 87, in forward
est_hm_list, encoding = self.net_hm(images)
File "C:\Users\ROHIT SANJAY\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'x'
any suggestions?
How can I predict my RGB images without camera parameters or 3D mesh data
Hi, guys
I find it is difficult to install opendr properly (I tried with anaconda python2, pip install, but failed).
I wonder if it is crucial for training? or it is just for visualization?
Thanks.
Hi @geliuhao ,
The reported AUC
on RHD
dataset, How you treat both hands in one image, did you consider both available hands in images for training and testing or have you chosen the more dominant hand from each image?
In your paper ,you said you can get the joint rotation from the joint location via solve an inverse kinematics, is there any corresponding part in your code?
Hi, I can't fins 3d hand mesh of your real-world-dataset in /data/real_world_testset, Can I have access to these files?
Thanks!
Hello ~
Thank you for your great jobs !
Could you provide me with data for training?
Thanks for your amazing idea,I train a YOLOv4 model to detect hand and want to run your project using a USB camera,But I don't known how to feed a value pose_root,can you explain how to calculate this value?
@3d-hand-shape For rotation , after rotating the image, you change the 3D annotation or camera parameter ?
Originally posted by @Ivamcoder in #10 (comment)
Hi, when I run the "python eval_script.py --config-file "configs/eval_real_world_testset.yaml" command I get this error: RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function) . Any idea on how to fix this?
Thanks in advance.
K.
Hi, I was wondering whether you might consider releasing the training code, which can increase your work's citations (since more people would base their research on yours), and contribute to the community. Thank you!
How can I predict my RGB images
bbox used for croping image is Ground Truth
or is from a detector ? and reported result in paper are according Ground Truth
hand area?
hand-graph-cnn-master/hand_shape_pose/util/renderer.py", line 223
color = color_list[color_id % len(color_list)]
TypeError: 'dict_values' object does not support indexing
Or is there a default setting so that the system can run on pure color input without any other information? Thank you very much.
Hi,
You have mentioned that in your dataset the unit of measurement is centimeters
. Is this the correct unit ?
I believe that it could be millimeter
due to following observation.
Observation description
I convert the global location to the hand joints depth. Now I took the relative depth from wrist joint i.e subtracting depth of joint i from joint wrist.
x' = depth_wrist - x
Note : all the units are converted cm
to mm
.
Issue I am thinking of is
mm
.Could you please confirm ? I might be wrong.
Having the same type of visualisation for Interhand dataset have different results. ( For the sake of clarity, unit in interhand is given as mm
which I got to confirm from creator of dataset.)
Further info :
I'm trying this on Windows 10 using a Git.
But I've faced a problem in the step of running code ' python eval_script.py --config-file "configs/eval_real_world_testset.yaml" '
And the error message indicated that it occurs on 'import opendr.camera'
So I've tried installing it for 'pip install' , 'conda install' and I'm trying opendr Github.. But I didn't find solution.
Could you give a help? what should I do.
And also I'm wondering whether I can use this on Webcam or other my video data !
How to get the GT boxes? Thank you. @Ivamcoder @3d-hand-shape
Originally posted by @prismformore in #8 (comment)
If I'm not sure the hand in picture is left or right
What should I do, thank you very much!
in net_util.py
line 86:y=torch.mm(W,x)
RuntTimeError:expected 'mat1' to be a CPU tensor,but got a CUDA tensor
The trainning script is available?
Hello !
First of all congratulations for your great jobs !
I tried to run the eval_script.py with exactly the same command lines of your tutorial, but the evaluated shape is not rendered on the images saved by the script. I checked the output of the mesh rendering step, and got a bunch of 0 in the rendered images.
I'm wandering if this is because of incorrect requirements' versions, but I'm not quite sure about this. So by the way, what are the exact requirements version of your code ? It seems that your pkl file are saved with python3, while opendr 0.76 do not support installations with python3.
Being grateful while waiting for your response !
Hi I am trying to implementing your pre-trained model on wild images, I noticed that cam_parameters & pose_roots are read from params.mat. May I know how could you calculate the pose_root?
Best
Tian
it is impossible that "setup.py" is not working
Thanks in advance,
Would you please help me to obtain this figure in your paper,
for ploting this curve , Do you use a template ? How you get the results from other works?
Is it possible to provide access to your code for this part of evaluation for fair comparison?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.