isk03276 / learntomoveur3

Learning various robotic manipulations tasks of the UR3

UR3-Deep-Reinforcement-Learning

You will be able to learn various tasks of the UR3 with robotiq85 gripper robot.
Learning method is based on the DRL(Deep Reinforcement Learning).
In this repo, we use CoppeliaSim (previously called V-REP), Pyrep.
Tasks

• DRL framework : rllib

• Supported tasks

  Tasks	Learned Task Example	Learning Curve
  reach
  TO DO	-	-

Install

This repo was tested with Python 3.7.9 version.

Coppeliasim

PyRep requires version 4.1(other versions may have bugs) of CoppeliaSim. Download:

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04

Add the following to your ~/.bashrc file: (NOTE: the 'EDIT ME' in the first line)

export COPPELIASIM_ROOT=EDIT/ME/PATH/TO/COPPELIASIM/INSTALL/DIR
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$COPPELIASIM_ROOT
export QT_QPA_PLATFORM_PLUGIN_PATH=$COPPELIASIM_ROOT

PyRep

Once you have downloaded and set CoppeliaSim, you can install PyRep: Move to home workspace

git clone https://github.com/stepjam/PyRep.git
cd PyRep
pip install -r requirements.txt
pip install -e .

Remember to source your bashrc (source ~/.bashrc) or zshrc (source ~/.zshrc) after this.

LearnToMoveUR3

Move to home workspace Clone repo and Install the python library:

git clone https://github.com/isk03276/LearnToMoveUR3.git
cd LearnToMoveUR3
pip install -r requirements.txt

Getting Started

python main.py --env-id ENV_ID --load-from MODEL_CHECKPOINT_PATH #Train
python main.py --env-id reach --test --load-from MODEL_CHECKPOINT_PATH #Test

Use Pretrained Model

python main.py --env-id reach --load-from pretrained_models/reach --test

learntomoveur3's Issues

QMutex: destroying locked mutex

"Hello, when I use the command to run the pre-trained model you provided, the following issue occurs. Could you please tell me the reason for this?

python main.py --env-id reach --load-from pretrained_models/reach/checkpoint_000575/checkpoint-575 --test

error
2023-12-11 17:22:15,948 WARNING ppo.py:395 -- train_batch_size (5000) cannot be achieved with your other settings (num_workers=4 num_envs_per_worker=1 rollout_fragment_length=100)! Auto-adjusting rollout_fragment_length to 1250.
2023-12-11 17:22:15,949 INFO ppo.py:415 -- In multi-agent mode, policies will be optimized sequentially by the multi-GPU optimizer. Consider setting simple_optimizer=True if this doesn't work for you.
2023-12-11 17:22:15,949 INFO trainer.py:906 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
(RolloutWorker pid=11459) 2023-12-11 17:22:18,686 WARNING env.py:136 -- Your env doesn't have a .spec.max_episode_steps attribute. This is fine if you have set 'horizon' in your config dictionary, or soft_horizon. However, if you haven't, 'horizon' will default to infinity, and your environment will not be reset.
(RolloutWorker pid=11462) 2023-12-11 17:22:18,719 WARNING env.py:136 -- Your env doesn't have a .spec.max_episode_steps attribute. This is fine if you have set 'horizon' in your config dictionary, or soft_horizon. However, if you haven't, 'horizon' will default to infinity, and your environment will not be reset.
(RolloutWorker pid=11461) 2023-12-11 17:22:18,715 WARNING env.py:136 -- Your env doesn't have a .spec.max_episode_steps attribute. This is fine if you have set 'horizon' in your config dictionary, or soft_horizon. However, if you haven't, 'horizon' will default to infinity, and your environment will not be reset.
(RolloutWorker pid=11460) 2023-12-11 17:22:18,748 WARNING env.py:136 -- Your env doesn't have a .spec.max_episode_steps attribute. This is fine if you have set 'horizon' in your config dictionary, or soft_horizon. However, if you haven't, 'horizon' will default to infinity, and your environment will not be reset.
2023-12-11 17:22:19,701 WARNING util.py:65 -- Install gputil for GPU system monitoring.
2023-12-11 17:22:19,708 INFO trainable.py:589 -- Restored on 172.25.216.219 from checkpoint: pretrained_models/reach/checkpoint_000575/checkpoint-575
2023-12-11 17:22:19,708 INFO trainable.py:597 -- Current state after restoring: {'_iteration': 575, '_timesteps_total': None, '_time_total': 24399.69826555252, '_episodes_total': 19362}
2023-12-11 17:22:21,457 WARNING deprecation.py:47 -- DeprecationWarning: compute_action has been deprecated. Use Trainer.compute_single_action() instead. This will raise an error in the future!
[Test] 1 reward -0.01 len 85.00, success mean 1.00
[Test] 2 reward -0.01 len 60.00, success mean 1.00
[Test] 3 reward -0.01 len 78.00, success mean 1.00
[Test] 4 reward -0.01 len 300.00, success mean 0.75
[Test] 5 reward -0.01 len 85.00, success mean 0.80
[Test] 6 reward -0.01 len 48.00, success mean 0.83
[Test] 7 reward -0.01 len 95.00, success mean 0.86
[Test] 8 reward -0.01 len 98.00, success mean 0.88
[Test] 9 reward -0.02 len 51.00, success mean 0.89
[Test] 10 reward -0.01 len 35.00, success mean 0.90
QMutex: destroying locked mutex

`ray.rllib.agents.callbacks` has been deprecated. Use `ray.rllib.algorithms.callbacks` instead.

(https://user-images.githubusercontent.com/118107900/229572253-90f571ad-cea9-4949-80e7-06e745f1f3bc.png)
Hi I have install all the libraries following the readme file. (Ray==2.3.1). But Following error appears. Can you please share which version of ray did u use. Or guide me what I am doing wrong here.

Typeerror= reduce ex error

File "/usr/lib/python3/dist-packages/yaml/representer.py", line 317, in represent_object
reduce = data.reduce_ex(2)
TypeError: reduce_ex() takes exactly one argument (0 given)

Hi, i've got an error when i started ur project. How can i fix it .
Thanks.
My python version 3.8.10

Package Version

Hello friend, first of all congratulate you for the great work published.
now my question is if you can leave a detailed list of the versions of the packages used as well as what version of ubuntu and ROS they were tested on.

Thank you very much for your answer