skumra / baxter-pnp

Baxter Pick and Place using GR-ConvNet. IROS 2020.

Baxter Pick and Place

This repository contains the implementation of the control module from the paper:

Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network

Sulabh Kumra, Shirin Joshi, Ferat Sahin

arxiv | video

If you use this project in your research or wish to refer to the baseline results published in the paper, please use the following BibTeX entry:

@article{kumra2019antipodal,
  title={Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network},
  author={Kumra, Sulabh and Joshi, Shirin and Sahin, Ferat},
  journal={arXiv preprint arXiv:1909.04810},
  year={2019}
}

Prerequisites

• A developer workstation meeting the minimum system requirements is available for use.
• The developer workstation has been set up successfully with Ubuntu and ROS - instructions here.
• You have configured your Development Workstation to access Github. Instructions here.
• A grasp generator. Our implementation of the GR-ConvNet based grasp generator is available at: https://github.com/skumra/robotic-grasping

Installation

• Download and Install the SDK dependencies

$ sudo apt-get install python-wstool python-rosdep

• Create a new catkin workspace.

$ mkdir -p ~/ros_ws/src
$ cd ~/ros_ws/src

• Download and copy the rosinstall file to the root of your Catkin workspace using wstool.

$ wstool init .
$ wstool merge https://raw.github.com/RethinkRobotics/baxter/master/baxter_sdk.rosinstall
$ wstool update
$ cd ..

This pulls down the development branches from all of our repositories into your source directory.

For Baxter Gazebo simulation, checkout the baxter_simulator repository:

$ cd ~/ros_ws/src
$ git clone https://github.com/RethinkRobotics/baxter_simulator.git

• Check out the Baxter PNP package into your ROS workspace

$ cd ~/ros_ws/src
$ git clone https://github.com/skumra/baxter-pnp.git

• Source ros setup if you haven't yet

$ source /opt/ros/kinetic/setup.bash

• Build and Install

$ cd ~/ros_ws
$ catkin_make
$ catkin_make install

• Use the baxter.sh script for proper environment setup

$ cp src/baxter/baxter.sh .

Run PNP in Simulation

• Run the baxter.sh script with sim specified

$ cd ~/ros_ws
$ ./baxter.sh sim

• Start simulation with controllers

$ roslaunch baxter_gazebo baxter_world.launch

• Start pick and place

$ cd ~/ros_ws
$ source devel/setup.bash
$ rosrun baxter-pnp run_pnp.py

Run PNP on Robot

• Run the baxter.sh

$ cd ~/ros_ws
$ ./baxter.sh

• Start pick and place

$ cd ~/ros_ws
$ source devel/setup.bash
$ rosrun baxter-pnp run_pnp.py

baxter-pnp's Issues

cmake failure

I had a problem with cmake

Question about grasp position info transfer

    homedir = os.path.join(os.path.expanduser('~'), "grasp-comms")
    self.grasp_request = os.path.join(homedir, "grasp_request.npy")
    self.grasp_available = os.path.join(homedir, "grasp_available.npy")
    self.grasp_pose = os.path.join(homedir, "grasp_pose.npy")

I am a new guy and I don't understand something about your code.
Did you convert the grasp infomation such as pick position generated from grconvnet to .npy format files?
And in baxter-pnp, you read these files to get the grasp info so that the robot can finish the pick and place mission.

If not, I want to know how do you implement the connection between the output of your network and the input of your robot.

thanks you for your work.

issue

When I got to catkin_make,terminal error:
/home/fang/ros_ws/src/baxter_simulator/baxter_sim_kinematics/src/arm_kinematics.cpp:248:65: error: conversion from ‘urdf::LinkConstSharedPtr {aka std::shared_ptr}’ to non-scalar type ‘boost::shared_ptr’ requested
boost::shared_ptr link = robot_model.getLink(tip_name);
/home/fang/ros_ws/src/baxter_simulator/baxter_sim_kinematics/src/arm_kinematics.cpp:255:60: error: no match for ‘operator=’ (operand types are ‘boost::shared_ptr’ and ‘urdf::JointConstSharedPtr {aka std::shared_ptr}’)
joint = robot_model.getJoint(link->parent_joint->name);

[rospack] Error: package 'baxter-pnp' not found

Setting environment variables still can't find，
Thanks
Thanks
Thanks

run run_pnp.py to simulate?

Hi! Thanks for your repository!
I have a camera D435i, but I don't have a robot, can I only do simulate with the Realsense camera in Gazebo?
Thanks in advance!

Could this be applied on ROS Melodic?

Can this be applied on ROS Melodic, which can be installed on Ubuntu18.04? The ROS Kinetic will no longer be supported soon according to http://wiki.ros.org. When I ran the code catkin_make, with Ubuntu18.04 and ROS Melodic, I got this:

Base path: /media/ghost/start/zyc/GR-ConvNet/ros_ws
Source space: /media/ghost/start/zyc/GR-ConvNet/ros_ws/src
Build space: /media/ghost/start/zyc/GR-ConvNet/ros_ws/build
Devel space: /media/ghost/start/zyc/GR-ConvNet/ros_ws/devel
Install space: /media/ghost/start/zyc/GR-ConvNet/ros_ws/install
####
#### Running command: "cmake /media/ghost/start/zyc/GR-ConvNet/ros_ws/src -DCATKIN_DEVEL_PREFIX=/media/ghost/start/zyc/GR-ConvNet/ros_ws/devel -DCMAKE_INSTALL_PREFIX=/media/ghost/start/zyc/GR-ConvNet/ros_ws/install -G Unix Makefiles" in "/media/ghost/start/zyc/GR-ConvNet/ros_ws/build"
####
-- Using CATKIN_DEVEL_PREFIX: /media/ghost/start/zyc/GR-ConvNet/ros_ws/devel
-- Using CMAKE_PREFIX_PATH: /media/ghost/start/zyc/GR-ConvNet/catkin_ws/devel;/opt/ros/melodic
-- This workspace overlays: /media/ghost/start/zyc/GR-ConvNet/catkin_ws/devel;/opt/ros/melodic
-- Found PythonInterp: /usr/bin/python2 (found suitable version "2.7.17", minimum required is "2") 
-- Using PYTHON_EXECUTABLE: /usr/bin/python2
-- Using Debian Python package layout
-- Using empy: /usr/bin/empy
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /media/ghost/start/zyc/GR-ConvNet/ros_ws/build/test_results
-- Found gtest sources under '/usr/src/googletest': gtests will be built
-- Found gmock sources under '/usr/src/googletest': gmock will be built
-- Found PythonInterp: /usr/bin/python2 (found version "2.7.17") 
-- Using Python nosetests: /usr/bin/nosetests-2.7
-- catkin 0.7.29
-- BUILD_SHARED_LIBS is on
-- BUILD_SHARED_LIBS is on
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 16 packages in topological order:
-- ~~  - baxter_common (metapackage)
-- ~~  - baxter_description
-- ~~  - baxter_sdk (metapackage)
-- ~~  - baxter_simulator (metapackage)
-- ~~  - rethink_ee_description
-- ~~  - baxter_maintenance_msgs
-- ~~  - baxter_core_msgs
-- ~~  - baxter_interface
-- ~~  - baxter_sim_controllers
-- ~~  - baxter_sim_io
-- ~~  - baxter_tools
-- ~~  - baxter_gazebo
-- ~~  - baxter_sim_kinematics
-- ~~  - baxter_sim_hardware
-- ~~  - baxter_sim_examples
-- ~~  - baxter_examples
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin metapackage: 'baxter_common'
-- ==> add_subdirectory(baxter_common/baxter_common)
-- +++ processing catkin package: 'baxter_description'
-- ==> add_subdirectory(baxter_common/baxter_description)
-- +++ processing catkin metapackage: 'baxter_sdk'
-- ==> add_subdirectory(baxter/baxter_sdk)
-- +++ processing catkin metapackage: 'baxter_simulator'
-- ==> add_subdirectory(baxter_simulator/baxter_simulator)
-- +++ processing catkin package: 'rethink_ee_description'
-- ==> add_subdirectory(baxter_common/rethink_ee_description)
-- +++ processing catkin package: 'baxter_maintenance_msgs'
-- ==> add_subdirectory(baxter_common/baxter_maintenance_msgs)
-- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy
-- baxter_maintenance_msgs: 7 messages, 0 services
-- +++ processing catkin package: 'baxter_core_msgs'
-- ==> add_subdirectory(baxter_common/baxter_core_msgs)
-- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy
-- baxter_core_msgs: 25 messages, 4 services
-- +++ processing catkin package: 'baxter_interface'
-- ==> add_subdirectory(baxter_interface)
-- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy
-- +++ processing catkin package: 'baxter_sim_controllers'
-- ==> add_subdirectory(baxter_simulator/baxter_sim_controllers)
-- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy
-- Could NOT find effort_controllers (missing: effort_controllers_DIR)
-- Could not find the required component 'effort_controllers'. The following CMake error indicates that you either need to install the package with the same name or change your environment so that it can be found.
CMake Error at /opt/ros/melodic/share/catkin/cmake/catkinConfig.cmake:83 (find_package):
  Could not find a package configuration file provided by
  "effort_controllers" with any of the following names:

    effort_controllersConfig.cmake
    effort_controllers-config.cmake

  Add the installation prefix of "effort_controllers" to CMAKE_PREFIX_PATH or
  set "effort_controllers_DIR" to a directory containing one of the above
  files.  If "effort_controllers" provides a separate development package or
  SDK, be sure it has been installed.
Call Stack (most recent call first):
  baxter_simulator/baxter_sim_controllers/CMakeLists.txt:4 (find_package)


-- Configuring incomplete, errors occurred!
See also "/media/ghost/start/zyc/GR-ConvNet/ros_ws/build/CMakeFiles/CMakeOutput.log".
See also "/media/ghost/start/zyc/GR-ConvNet/ros_ws/build/CMakeFiles/CMakeError.log".
Invoking "cmake" failed

I'm not sure if it arose out of the wrong distribution of ROS or not. But there had been no errors when I ran catkin_make in a project which was initialized using catkin_init_workspace.
Thanks for your work!

Missing package.xml and CMakelists.txt

I'm trying to use this repository on a real Baxter robot. However it seems that the package.xml and CMakeLists.txt are missing. As such, the package is not built by catkin. Could you provide these files please?
Thank you very much.