luigiberducci / auto-shaping Goto Github PK

create basic documentation to explain the goal of the library, its methods, and how to use it.

Examples:

• how to use default configurations in standard environments
• how to use a custom specification
• how to train with RL libraries (sb3, omnisafe)

For each example, either create python script or jupyter notebook.
Comment code and explain it.

Benchmark of all the shaping methods in a set of standard environments

Environments:

• Cartpole
  • ⬛ ⬛ ⬛ Default
  • ⬛ ⬛ ⬛ TLTL
  • ⬛ ⬛ ⬛ BHNR
  • ⬛ ⬛ ⬛ HPRS
• Bipedal Walker
  • ⬛ ⬛ ⬛ Default
  • ⬛ ⬛ ⬛ TLTL
  • ⬛ ⬛ ⬛ BHNR
  • ⬛ ⬛ ⬛ HPRS
• Bipedal Walker Hardcore
  • ⬛ ⬛ ⬛ Default
  • ⬛ ⬛ ⬛ TLTL
  • ⬛ ⬛ ⬛ BHNR
  • ⬛ ⬛ ⬛ HPRS
• Lunar Lander
  • ⬛ ⬛ ⬛ Default
  • ⬛ ⬛ ⬛ TLTL
  • ⬛ ⬛ ⬛ BHNR
  • ⬛ ⬛ ⬛ HPRS

For each of them, use specifications from the HPRS paper.
Train for multiple seeds till convergence (k=3) and make the learning curves available on wandb.

Methods:

• TLTL
• BHNR
• HPRS

Each of them must be compatible with the specification language presented in the HPRS paper.

Open issues:

• How can we use in the reward spec variables that are not in the observed state? (e.g., x in bipedal walker)
• How can we compactly specify collections of variables in the spec? (e.g., lidar = [l1, l2, ..., l10] in bipedal walker)

• safety gym api
• integration omnisafe
• example omnisafe

• Policy Assessment Metric (PAM) from HPRS paper
• Rank Preserving Reward (RPR) from Receding Horizon Planning with Rulebook paper