jharwell / sierra Goto Github PK
View Code? Open in Web Editor NEWAutomation framework for the scientific method in AI research
License: MIT License
Automation framework for the scientific method in AI research
License: MIT License
Basically, increasing arena size while keeping area occupied by robots the
same.
Will be VERY helpful when trying to extrapolate scripted experiment overall
duration for MSI.
Right now I hardcode an estimate into the footbot model. Given that the estimate
will undoubtedly change between simulations/swarm sizes, I should probably get
it from the simulation results (or even from the max speed which is specified in
the simulation input file).
For each column in an averaged .csv, graphs should be generated showing the
distribution of the averaged data:
For some comparison graphs, I need the swarm size to start at 1 (scalability),
for some, it needs to start at 2 (blocks collected). Adding a switch/argument to
control this is necessary.
The homegrown csv_class class is pretty good, but pandas is more robust,
cleaner, etc. When switching, we need to maintain the checks for same number
rows/colums in .csv files before averaging that currently exist.
This has become a bottleneck when running on MSI, as average 50-100 simulations
can take upwards of an hour with the current method, which takes upwards of an
hour. It is only a few seconds with ~4 or so simulation runs, which is totally
fine.
Cache footprint, max size, usage penalty
This would enable me to generate a much larger set of experiments in a batch
from TWO criteria (say swarm size and # of blocks). Would really just be
generating the 2 lists of sets of changes, and then combining them such that the
resulting list of sets contains M*N changesets.
2018-12-18: Some progress has been made on this with the command line string
parsing + dynamic class generation, BUT I still can't use a batch
criteria of swarm_size + temporal_variance (for example).
2019-09-23: For now, limit this to at most 2 criteria, specified on the cmdline
separated by commas.
The current collision avoidance entry/exit/duration models, even for the super
simple stateless foraging in a square arena with random block distribution are
way off (entry) or incomplete/not implementation (exit, duration)
More accurate models for each of the 3 phases of CA should be developed so that
transition probabilities for MC models can be analytically estimated.
So that instead of specifying the directories for EACH and EVERY stage of the
pipeline, you can just specify the root directory and it will create everything
in there. You should still be able to specify specific directories if you want
to though.
sierra should be able to:
How to add a new controller to generators
How to add a new variable
How to add a new intra-exp graph
How to add a new inter-exp graph
How run on MSI (this is sort of already there, but needs a revamp).
Update the readme to explain what the purpose of each directory in sierra
is.
Update readme to say that sierra does not delete any directories for you,
which can cause problems if you shrink size of exp/# simulations when it comes
to averaging and displaying graphs.
I have linegraphs, but having histograms for SOME kinds of experiments
might be helpful (not for ones where I vary the swarm size, for example, but for
temporal variance they might provide insight).
No idea why--some setting in matplob lib. I need to be able to pick a color for
the error bars, as otherwise they appear as the same color as the line they
correspond to, which makes it hard to see/make sense of.
Given a set of csv file names, sierra should be able to read all them
in, and:
Average all columns together (except those given in an exception
list), and write the result to a final csv.
Compute the std deviation for each row in each column, and output
that in a second .csv.
Tied to fordyca issue.
In addition to being able to run N copies of a simulation and average the
results, I also need to be able to compare results of experiments with respect
to variation in some parameter in the input file (i.e. # of robots).
My thought is to create a batched mode/batched pipeline that will:
I haven't tackled graph generation yet, so that shouldn't be considered in
this feature.
For a log 256 criteria, an experiment is run with 512 robots, which is obviously
wrong.
Simple things that you want to graph within a single experiment from the
averaged data:
Should use the scalability measure as a template, as the process for gathering
the information it needs should be very similar.
Methods, always/never partition, as well as method parameters if applicable.
Mostly because I'm impatient.
This is for validating my analytical model against experimental data. Should
include:
Predicted # robots entering CA each timestep, given swarm size, arena
geometry, and robot kinematic parameters
Predicted # robots exiting CA each timestep, given swarm size, arena
geometry, and robot kinematic parameters
Predicted CA duration, given swarm size arena geometry, and robot kinematic
parameters
Given a final set of .csv files, for a set of N simulation runs and a
std deviation file, sierra should be able to generate a set of plots (probably with
matplotlib or similar) representing the simulation results.
The easiest method is factorial, where given a set of parameters and
ranges/values (probably in a JSON file), exhaustively test all
combinations. This will realistically only work for small sets of
parameters due to combinatorial explosion.
There are other ways of traversing the parameter space, and those
should be incorporated as well (i.e. hook into PyPet).
The framework would need to be able to run N copies of a particular
simulation scenario on a single computer (serially), as well as on a cluster in a
distributed fashion (simultaneously).
For distributed execution, definitely want to hook into something like
cosmos, luigi, sciluigi, etc. that already exists--no need to reinvent
the wheel.
This would be for 3D comparisons (e.g. X=#robots, Y=#blocks in arena, Z=Total
blocks gathered during simulation).
This is only necessary when running with depth1, caches, and random block
distribution, but it is VERY necessary then.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.