GithubHelp home page GithubHelp logo

Comments (4)

maxspahn avatar maxspahn commented on August 18, 2024

Currently, if a new metric needs to be added, the name of the metric must be added at various places in the code. This process can be made simpler.

https://github.com/maxspahn/localPlannerBench/blob/d2e5f47acd8ddcb2bdca362f04c3258991510d31/plannerbenchmark/exec/postProcessor#L30

https://github.com/maxspahn/localPlannerBench/blob/d2e5f47acd8ddcb2bdca362f04c3258991510d31/plannerbenchmark/exec/postProcessor#L180

from localplannerbench.

niketagrawal avatar niketagrawal commented on August 18, 2024

Developer notes on the current implementation:

  • To create a new metric, inherit from the metric class and implement the computemetric() method in metrics.py
  • In order to use the metric for benchmarking purposes for a specific type of robot, it must be instantiated for that type of robot in the postprocessor script.
self._metrics['time2Goal'] = TimeToReachGoalMetric(
            "time2Goal",
            ["q0", "q1", "goal_0_0", "goal_1_0", "t"],
            {"m": 2, "des_distance": self._experiment.primeGoal().epsilon()},
        )
- The fist argument is a string to represent the type of metric. It doesn't need to be passed as an argument here, it can be an instance attribute of the metric class.
- The 2nd argument is a list of column indices in res.csv that needs to be selected for postprocessing. The columns chosen for each type of robot and metric combination will differ which requires a unique instantiation for each metric and robot combination, but this issue can be solved by renaming the columns in the res.csv to represent the parameters type and not the specific variable names. 
- The 3rd argument comes from experiemnt.py; this can be standardized too. in the above manner.
  • Requirements for creating a new metric

    • Knowledge of the computation logic for the metric; this is needed to implement the computemetric method()
    • Knowledge of how the new metric will be instantiated for a particular type of robot. This is needed to specify which column indices from res.sv are required for metric computation.
  • The arguments of the class instantiation of the metrics in the postprocessor script can be standardized so that they can be embedded in the metric class itself.

  • self.experiment.robot_type() can be used to construct 2nd and 3rd argument of the class instantiation

  • A new metric may not apply to all the robot types; it would depend on the what is being computed in the metric.

from localplannerbench.

niketagrawal avatar niketagrawal commented on August 18, 2024

Expected functionality (tentative): A metric class should be instantiated for a particular type of robot based on the arguments passed in the postprocessor command. Currently, all metric class are being instantiated for all types of robots in advance.

from localplannerbench.

maxspahn avatar maxspahn commented on August 18, 2024

This could be done with the registry pattern. Similar to #25 for planners.

from localplannerbench.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.