Code to reproduce the experiments in CUP: Constrained Update Projection Algorithms for Safety Robot Control
- Clone
CUP
git clone https://github.com/BellmanTimeHut/CUP.git
cd CUP
conda env create -f environment.yml
conda activate CUP
To get start with CUP, run:
python main.py --file-prefix 'Test' --env-id Ant-v3 --algo CUP --constraint velocity --cost-lim 103.12 --seed 1024
To obtain the results in our experiments, the following hyperparameters are the required.
Hyperparameter | CUP | PPO-L | TRPO-L | CPO | FOCOPS |
---|---|---|---|---|---|
No. of hidden layers | 2 | 2 | 2 | 2 | 2 |
No. of hidden nodes | 64 | 64 | 64 | 64 | 64 |
Activation | tanh | tanh | tanh | tanh | tanh |
Initial log std | -0.5 | -0.5 | -1 | -0.5 | -0.5 |
Discount for reward |
0.99 | 0.99 | 0.99 | 0.99 | 0.99 |
Discount for cost |
0.99 | 0.99 | 0.99 | 0.99 | 0.99 |
Batch size | 5000 | 5000 | 5000 | 5000 | 5000 |
Minibatch size | 64 | 64 | N/A | N/A | 64 |
No. of optimization epochs | 10 | 10 | N/A | N/A | 10 |
Maximum episode length | 1000 | 1000 | 1000 | 1000 | 1000 |
GAE parameter (reward) | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 |
GAE parameter (cost) | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 |
Learning rate schedule for Actor-Critic | True | False | False | False | True |
Learning rate for policy | N/A | N/A | |||
Learning rate for reward value net | |||||
Learning rate for cost value net | |||||
Learning rate for |
0.01 | 0.01 | 0.01 | N/A | 0.01 |
Clip gradient norm | True | False | False | False | True |
L2-regularization coeff. for value net | |||||
Clipping coefficient | N/A | 0.2 | N/A | N/A | N/A |
Damping coeff. | N/A | N/A | 0.01 | 0.01 | N/A |
Backtracking coeff. | N/A | N/A | 0.8 | 0.8 | N/A |
Max backtracking iterations | N/A | N/A | 10 | 10 | N/A |
Max conjugate gradient iterations | N/A | N/A | 10 | 10 | N/A |
Iterations for training value net | 1 | 1 | 1 | 80 | 1 |
Temperature |
1.5 | N/A | N/A | N/A | 1.5 |
Trust region bound |
0.02 | N/A | 0.01 | 0.01 | 0.02 |
Initial |
0, 2 | 0, 1 | 0, 2 | N/A | 0, 2 |
Hyperparameter | Default for MuJoCo and SafetyGym | (Swimmer-v3) | (HumanoidCircle-v0) |
---|---|---|---|
Critic grad norm (Reward) | 0.5 | 0.5 | 0.5 |
Critic grad norm (Cost) | 0.5 | 0.5 | 0.5 |
Actor grad norm in policy improvement | 0.5 | 0.5 | 0.5 |
Actor grad norm in projection | 0.5 | 10 | 10 |