jharwell / sierra Goto Github PK

Basically, increasing arena size while keeping area occupied by robots the
same.

• Would take in/have a robot area parameter

Will be VERY helpful when trying to extrapolate scripted experiment overall
duration for MSI.

Right now I hardcode an estimate into the footbot model. Given that the estimate
will undoubtedly change between simulations/swarm sizes, I should probably get
it from the simulation results (or even from the max speed which is specified in
the simulation input file).

For each column in an averaged .csv, graphs should be generated showing the
distribution of the averaged data:

• std deviation over time
• variance over time
• Mean over time

For some comparison graphs, I need the swarm size to start at 1 (scalability),
for some, it needs to start at 2 (blocks collected). Adding a switch/argument to
control this is necessary.

The homegrown csv_class class is pretty good, but pandas is more robust,
cleaner, etc. When switching, we need to maintain the checks for same number
rows/colums in .csv files before averaging that currently exist.

This has become a bottleneck when running on MSI, as average 50-100 simulations
can take upwards of an hour with the current method, which takes upwards of an
hour. It is only a few seconds with ~4 or so simulation runs, which is totally
fine.

Cache footprint, max size, usage penalty

This would enable me to generate a much larger set of experiments in a batch
from TWO criteria (say swarm size and # of blocks). Would really just be
generating the 2 lists of sets of changes, and then combining them such that the
resulting list of sets contains M*N changesets.

2018-12-18: Some progress has been made on this with the command line string
parsing + dynamic class generation, BUT I still can't use a batch
criteria of swarm_size + temporal_variance (for example).

2019-09-23: For now, limit this to at most 2 criteria, specified on the cmdline
separated by commas.

The current collision avoidance entry/exit/duration models, even for the super
simple stateless foraging in a square arena with random block distribution are
way off (entry) or incomplete/not implementation (exit, duration)

More accurate models for each of the 3 phases of CA should be developed so that
transition probabilities for MC models can be analytically estimated.

So that instead of specifying the directories for EACH and EVERY stage of the
pipeline, you can just specify the root directory and it will create everything
in there. You should still be able to specify specific directories if you want
to though.

sierra should be able to:

1. Parse a .argos input file
2. Given a path like "params.occupancy_grid.pheromone.rho" (maybe
   something a little different dependeng what xml python parsing
   looks like), it should be able to replace any value in the .argos
   file with any other value.

• How to add a new controller to generators

• How to add a new variable

• How to add a new intra-exp graph

• How to add a new inter-exp graph

• How run on MSI (this is sort of already there, but needs a revamp).

• Update the readme to explain what the purpose of each directory in sierra
  is.

• Update readme to say that sierra does not delete any directories for you,
  which can cause problems if you shrink size of exp/# simulations when it comes
  to averaging and displaying graphs.

I have linegraphs, but having histograms for SOME kinds of experiments
might be helpful (not for ones where I vary the swarm size, for example, but for
temporal variance they might provide insight).

No idea why--some setting in matplob lib. I need to be able to pick a color for
the error bars, as otherwise they appear as the same color as the line they
correspond to, which makes it hard to see/make sense of.

Given a set of csv file names, sierra should be able to read all them
in, and:

1. Average all columns together (except those given in an exception
   list), and write the result to a final csv.

2. Compute the std deviation for each row in each column, and output
   that in a second .csv.

Tied to fordyca issue.

In addition to being able to run N copies of a simulation and average the
results, I also need to be able to compare results of experiments with respect
to variation in some parameter in the input file (i.e. # of robots).

My thought is to create a batched mode/batched pipeline that will:

• Generate different "template" files from the original template
• Feed the generated template files to the existing experimental pipeline,
  which will generate the necessary simulation input files, run experiments,
  and average the results.

I haven't tackled graph generation yet, so that shouldn't be considered in
this feature.

For a log 256 criteria, an experiment is run with 512 robots, which is obviously
wrong.

Simple things that you want to graph within a single experiment from the
averaged data:

• block acquisition over time
• block transport over time
• distance traveled over time

Should use the scalability measure as a template, as the process for gathering
the information it needs should be very similar.

Methods, always/never partition, as well as method parameters if applicable.

Mostly because I'm impatient.

This is for validating my analytical model against experimental data. Should
include:

• Predicted # robots entering CA each timestep, given swarm size, arena
  geometry, and robot kinematic parameters

• Predicted # robots exiting CA each timestep, given swarm size, arena
  geometry, and robot kinematic parameters

• Predicted CA duration, given swarm size arena geometry, and robot kinematic
  parameters

Given a final set of .csv files, for a set of N simulation runs and a
std deviation file, sierra should be able to generate a set of plots (probably with
matplotlib or similar) representing the simulation results.

The easiest method is factorial, where given a set of parameters and
ranges/values (probably in a JSON file), exhaustively test all
combinations. This will realistically only work for small sets of
parameters due to combinatorial explosion.

There are other ways of traversing the parameter space, and those
should be incorporated as well (i.e. hook into PyPet).

The framework would need to be able to run N copies of a particular
simulation scenario on a single computer (serially), as well as on a cluster in a
distributed fashion (simultaneously).

For distributed execution, definitely want to hook into something like
cosmos, luigi, sciluigi, etc. that already exists--no need to reinvent
the wheel.

• Too large/does too many things at the moment. Need to break into its
  constituent pipeline stages.

This would be for 3D comparisons (e.g. X=#robots, Y=#blocks in arena, Z=Total
blocks gathered during simulation).

This is only necessary when running with depth1, caches, and random block
distribution, but it is VERY necessary then.