[Error] [carb.gym.plugin] Gym cuda error: out of memory: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 1721 about unidexgrasp

[wkwan7]

Does this error occur during visual-based policy training? For visual-based policy training, It requires a relatively large GPU memory, compared to state-based policy. I remember we set num_envs to 16 on RTX 3090 (~24GB).

[lptl]

Hi, yes, it is visual-based policy training.

[lptl]

Hi, may I ask how many epochs have you trained for the vision-based policy training? I am currently using 10,000 and don't see any effect.

[wkwan7]

Assuming you are using a vision-based RL (PPO) approach, as pointed out in our paper, directly training a vision-based RL policy doesn't work (although we do provide this option in our code). If you wish to train a vision-based policy, our recommendation is to first successfully train a state-based policy and then follow our code to perform state-to-vision policy distillation using DAgger.

[lptl]

Hi, may I ask how the dexgrasp_policy/meshdata_scaled and dexgrasp_policy/meshdata_pc_feat come from? I can't find the groundtruth scale. Thanks!

[mzhmxzh]

We just uploaded the data here. You can unpack them and put them under the directory dexgrasp_policy/assets.

[lptl]

Hi, thanks! I discovered that there are only core and sem folder in meshdatav3_pc_feat.zip without mujoco and ddg. And may I ask where the scale in the object_code_list in configuration yaml files comes from?

[wkwan7]

Hello! The meshdatav3_pc_feat stores features from the pre-trained category label classification task. Since objects in mujoco and ddg do not have defined "category labels", they were not included in the pre-training. If these objects are used in training, we simply set all their features to 0. (Perhaps you can use a better pre-training task). Regarding scale, we simply applied filtering to all object scales, filtering out objects (with different scales) that are too large or too small (actually it's very challenging for a single training policy to handle these objects). I will update the object/scale data we used to the repo next week if you need it.

[lptl]

hi, thanks for your help of the dataset and the object/scale information! The object/scale data would be really helpful.

[lptl]

And may I ask whether there exists any other things to notice when training state-based goal-conditioned case? I changed the config file from goal-cond:false to goal-cond:true, and the reward is almost only around 20, which makes the success rate is 0.0.

[wkwan7]

Hello! I apologize for the delayed response. 1. Regarding the strategy training issue: Due to the large number of objects, we did not test "train from scratch" for all individual objects in both goal-conditioned and non-goal-conditioned scenarios (in fact, quite a few objects would fail when "train from scratch"). However, as pointed out in our paper, "curriculum learning" is crucial. My suggestion is to first train a non-goal-conditioned model on a simple object (for example, data 'sem/Car-669043a8ce40d9d78781f76a6db4ab62':[0.06] which we use for the training at first), and then fine-tune this model on more challenging objects or in the goal-conditioned scenario. 2. Concerning the object/scale data, we are currently looking for a better method to select scales using filters. It will take some time before we can fully organize and release it. If you need it, I can send you the current temporary version via email. If you have any other questions, feel free to contact us.

[lptl]

Thank you so much for your reply! Expect for the release of the object-scale list. It would be really helpful if you can send me the current version via email, my email is [email protected]. Thank you very much!

[mxllc]

In the paper, the treatment of the non-goal-conditioned model involves removing the goal input and goal reward during RL training. However, in the shadow_hand_grasp.py, I only see that the goal reward has been removed, and the goal input is still used as an input to the network. I find this somewhat puzzling.

UniDexGrasp/dexgrasp_policy/dexgrasp/tasks/shadow_hand_grasp.py

Lines 797 to 802 in e724a94

	# 207:236 goal
	hand_goal_start = obj_obs_start + 16
	self.obs_buf[:, hand_goal_start:hand_goal_start + 3] = self.delta_target_hand_pos
	self.obs_buf[:, hand_goal_start + 3:hand_goal_start + 7] = self.delta_target_hand_rot
	self.obs_buf[:, hand_goal_start + 7:hand_goal_start + 29] = self.delta_qpos

[wkwan7]

Hello, thank you for pointing out this issue! When training with non-goal-conditioned, we do not use goal-reward and will simply set these three lines to a value of 0.
https://github.com/PKU-EPIC/UniDexGrasp/blob/e724a94f8260dee888477a2e9d048272dfe4fd7c/dexgrasp_policy/dexgrasp/tasks/shadow_hand_grasp.py#L799C9-L801C84

[lptl]

Hi, May I ask how to extract the object's pose and rotation from the npz file of datasetv4.1? I currently extracted the target hand pose and rotation, as the code shows. I wonder how to find the pose and rotation of the object so that the object is grasped in the hand. Thanks!