The variables n and m are used in multiple locations without being explained.
Make sure to either document that, or change the variable naming.

Version v.1.0.0 of the package is available on PyPi, but to add a custom planner currently you have to change stuff within the package. Either placing your custom planner in the planner folder or importing your custom planner manually in the runner script in a different way.

One option would be to use an environment variable that contains the location of the custom planners. Within runner we can use this variable to import (and register through the meta class) the custom planners.

🚀 Feature Request

Include customizable uncertainty to help make the benchmarks more representative of real scenarios.

Motivation

At this moment the planner will have access to the ground truth state and observation (in the form of obstacle positions) from the physics simulator. However, in the real world this will not be the case. Including customizable uncertainty could help make the benchmarks more representative of real scenarios.

Especially in planar cases (mobile robots, cars), it is very useful to have a more involved model of moving obstacles.
Therefore, the integration of a social forces model would be beneficial. An implementation as suggested in https://github.com/yuxiang-gao/PySocialForce/tree/dev seems relatively simple to add.

🚀 Feature Request

Add interface to global planners found in OMPL, that give guidance to the local planners.

Motivation

Global planners like found in OMPL can influence the performance of local planners, depending on how they are affected by the guidance. Including an option to interface with global motion planners and test the whole pipeline could be a valuable enhancement.

• Provide information on how to run the examples. Mention the pre-requisities, for example, if a licensed software is required.
• Specify the installation pre-requisites before the installation instructions to have a logical continuity.
• Provide installation instructions for different Operating systems if applicable.

For example,

Getting started

Pre-requisites

• Python 3.x

• Poetry: Install Poetry using ...

• Linux (if software is Linux only)

Installation

1. Install the software package using the command ...

Include any command to test the installation of the software if possible. Screenshots showing the output of each command would also be helpful.

Tutorials

Run experiments ...

🚀 Feature Request

Include a wrapper to the currently considered state-of-the-art in local motion planning.

Motivation

To gain more attention and traction to localPlannerBench, we should consider implementing wrappers ourselves for the state-of-the-art planners.

Example candidates:

• https://github.com/UM-ARM-Lab/pytorch_mppi
• https://github.com/snape/RVO2
• https://github.com/svenkreiss/socialforce (not the original social forces but repository looks nice, so maybe?)
• https://github.com/vita-epfl/trajnetplusplusbaselines
• ...

Leave a comment with link, if you know more!

Running an experiment with a point mass urdf, two-dimensional goal and fabrics results in a key error when post-processing.
Variables for the goals and forward kinematics are created based on the dimensions of the obstacles, which is often three-dimensional.

If the goal is defined to be two-dimensional and both the goal and forward kinematics are stored as two-dimensional, the postProcessor will try to find the third-dimension variables for both the goal and forward kinematics, which does not always exist.

Solution: create the variable names for goal and forward kinematics dependent on the goal indices instead of the obstacle dimensions.

Make sure that basic functionality is not destroyed by changes made during structure improvement.
This can be done with GitHub actions and a CI.

Some of the methods such as name are defined as class attributes with leading underscore. This is a convention for private variables, but does not actually make the variable private. Consider using property decorator, https://www.geeksforgeeks.org/python-property-decorator-property/

Currently the GNUplot scripts within post_processing contains hardcoded names for planners to create proper labels.

Ideally this should be parameterized, so any new planner automatically has the right label.

Installing development-only dependencies such as pytest, black and pylint conflicts with the version of Python pinned in pyproject.toml
(python = '^3.6,<3.11'), for example, latest version of black needs python >3.6.2

Allow to only plot results again, without postprocessing again.

Keep documentation in mind when working on #30

Running the example from the tutorial results in a key error for me when running the postProcessor after installing the repository with Poetry on Python 3.8.10.

The key interval can't be found, which gives an error in caseEvaluation.py at line 40.
Adding interval: 1 to the planner.yaml seems to fix the issue, I used 1 as found as a default value in the documentation.

When running the example in a fresh install some dependencies are missing:
gnuplot
Library GLU option

Also, some error in the documentation: pdbplanner must be replaced with pdplanner.

The planners are set every run. This can result in errors if a planner is not reset. The fabrics planner, for example, does not reset its obstacles, which results in an error where the obstacles are being declared twice at a second run.

A solution would be to set the planner before the while loop, or to reset the obstacles in the fabrics planner.

When rerunning an experiment from a study with multiple planners, the result is not always exactly the same, it seems that the initial state is not correctly reset.
I ran a study with multiple fabric planners, rerunning the planner that is executed first, gives exactly the same results.
However, when rerunning the second or one of the following planners, gives slightly different results.

This might be due to line 172 below, which is declared before looping over the planners.
https://github.com/maxspahn/localPlannerBench/blob/a728b3b4ab93e04ea2786f312575aaa555dbe2fb/plannerbenchmark/exec/runner.py#L172-L177

Not sure, but it seems that setting line 172 in between 176 and 177 solves the issue.

I think the metrics are good candidates to create some unit tests for, since it should be clearly defined input->output. Especially if we want people to add their own metrics, we should give some examples of how to unit test.

The readme misses documentation on how to run the postprocessing step.

Options:

• Convert Mpc forcespro planner to an opensource solver (acado / fmin python / scipy / etc.)
• Create closed (binary) version of FabricPlanner.

Currently, the wrapper to yaml-files is included into the constructor method for experiments and planner.
This should be separated.

Potentially, use a singleton pattern and json files instead of yaml: https://linuxhint.com/yaml-vs-json-which-is-better/

Some classes, experiment.py, mpcPlanner.py, fabricPlanner.py are mainly parsing yaml files into dictonaries.
Then individual getters are created for the key-value pairs. This is responsible for large parts of the code.

https://github.com/maxspahn/localPlannerBench/blob/d2e5f47acd8ddcb2bdca362f04c3258991510d31/plannerbenchmark/planner/fabricPlanner.py#L56-L89

Maybe, it is possible to switch to a different parameter storing style, see python dataclasses.

Work from bottom dependencies upwards, to avoid conflicts.

🚀 Feature Request

Add support for observations in the form of occupancy maps, obstacle semantics, semantic maps, or even direct camera / lidar data.

Motivation

Recent local motion planning methods aim to base their motion on observations that contain more information, sometimes even direct sensor data. To include these methods we should consider adding support for more fidelity in the observations.

There are a couple of vulnerabilites in the package.
5/6 can be fixed by updating pillow to 9.0.1. However, this implies an upgrade to at least python3.7.

Consider using module-level, class-level and function-level docstrings for describing the purpose of the module, description of the class or a function (especially when the function name and arguments can't convey what the function does).

Using the numpydoc documentation style for writing a docstring is recommended.

Required for function docstring:

• One line description of what the function does
• Description of the input parameters and their type
• Description of the return parameters and their type

Examples:

def multiplication_table(zero_out_multiples=None):
    """Creates a multiplication table.
    Returns a two-dimensional NumPy array that contains the multiplication 
    table from 1 to 12.  
    
    Parameters
    -----------
    zero_out_multiples : int, optional
        When this parameter is set to an integer number, then the 
        multiplication table that is returned by the function will have all
        multiples of the given number set to zero. The default value of this 
        parameter is None.
    Returns
    -------
    array : ndarray
        A two-dimensional NumPy array that contains the multiplication table 
        from 1 to 12 with the multiples of zero_out_multiples paramaters 
        passed as argument set to zero. 
    """