#19

Ref)

• HAC
  • Levy, Andrew, et al. "Learning multi-level hierarchies with hindsight." arXiv preprint arXiv:1712.00948 (2017).
  • code : Hierarchical-Actor-Critic-HAC-PyTorch
• UOF,
  • Yang, Xintong, et al. "Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation." IEEE Transactions on Neural Networks and Learning Systems (2021).
  • code : UOF-paper-code

Results:

ropiens/hierarchical-actor-critic-pytorch

to-do-list

• Implementation of DDPG + HER only. ropiens/universal-option-framwork-pytorch#3
• Implementation of high-level policy and DIOL. ropiens/universal-option-framwork-pytorch#4
• add AAES ropiens/universal-option-framwork-pytorch#5
• add Abstract Demonstration.

Results

• ropiens/universal-option-framwork-pytorch

Survey : the way to manipulate for block-stacking

Approach from TAMP

• Classical TAMP vs Learning-based TAMP
• Classical TAMP

Approach from Hierarchical RL

• nikhilbarhate99/Hierarchical-Actor-Critic-HAC-PyTorch
• Hierarchical Deep Reinforcement Learning (HDQN)
  
• Learning Multi-Level Hierarchies with Hindsight(HAC)
• Introduce to HRL
• Meta-Learning shared Hierarchy(MLSH)
• Article : The Promise of Hierarchical Reinforcement Learning

Multi-step Manipulation(for long-horizon tasks)

• A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects -> this paper just introduces to make subgoal planning by representation learning of objects pointcloud
  • its manipulation skills from here
• Hierarchical Reinforcement Learning with Universal Policies for Multi-Step Robotic Manipulation, https://www.youtube.com/watch?v=n_wQuf4r0qk
• Planning for Multi-Stage Forceful Manipulation

Multi-goal Task Approach(UMDP)

• Hindsight Experience Replay
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning

Approach from Relational RL

• Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning

Good Examples

• IanYangChina/pybullet_multigoal_gym, paper : https://arxiv.org/pdf/2105.05985.pdf
• qgallouedec/panda-gym -> pre-trained model on rl-baselines3-zoo

Other Interesting Works(Advanced Works)

• IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks, (https://clvrai.github.io/furniture)
• Learning to guide task and motion planning using score-space representation
• Self-play for goal discovery in robotics manipulation
• Transporter Network: Rearranging the Visual World for Robotic Manipulation
• CAVIN: Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
• Neural Task Programming: Learning to Generalize Across Hierarchical Tasks

Check this PR -> #17

Experiments for single-step task on panda-gym

• #4
• #5

Describe some reports for the experiments below.

Results

• PandaReach-v1, PandaPush-v1, PandaPickAndPlace-v1 -> converged well.
• but PandaStack-v1 -> never converged w/ SAC+HER agent only.

Description

When we run make format, all python files' code-style are changed.
We need some rules for excluding non-target files.
e.g. panda-gym and venv, etc

Let's start the project-sandwich-man
kick-off : 21/08/04

HIERARCHICAL POLICY LEARNING IS SENSITIVE TO
GOAL SPACE DESIGN

• https://tarl2019.github.io/assets/papers/dwiel2019hierarchical.pdf

Planning

• #10
• based on ropiens/hierarchical-actor-critic-pytorch
  • #11
  • #12(skip)
  • #13(skip)
• #14
  • #18
• #15
• #16