idsia / sacred Goto Github PK
View Code? Open in Web Editor NEWSacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.
License: MIT License
Sacred is a tool to help you configure, organize, log and reproduce experiments developed at IDSIA.
License: MIT License
In case of multiple ConfigScopes
we could allow using the non-JSON locals from one cfg in the others. This would ease implementing the network architecture module providing inputLayer
and outputLayer
for later usage in config scopes.
But it will complicate the implementation. Is it worth the trouble?
for example in
It would be very useful if one could add files to an experiment that can be then be stored by the observers alongside the information for the run.
I can think of several different types of files to add:
This includes additional source-files etc.. They would be part of the info about the experiment. So they could potentially be shared among many runs.
ex.add_dependency("/my/path/filname.foo")
Files that are written once and never changed, but belong to the run not to the experiment. For example a report that is written after the run is finished would fall in this category. They might be a special case of Dynamic Files.
Files that are updated continuously. So they should be updated as part of the heartbeat. Examples include current weights or log files.
run
object.f = run.create_tracked_file(filename_in_db)
run.track_file(filename, filename_in_db=None)
might be sufficient.f = run.get_tracked_file(filename_in_db)
that returns the file if present or otherwise creates it?Some people might prefer using regular dictionaries
so for example this will fail:
python hello_config.py with message="Hello World"
If you execute a command with a missing value some parameters with default values will be reported as missing too.
This would:
Problems might be:
Maybe also extract out host info?
If a config update from the command line added a new entry we should issue a warning.
If you try to add a config entry to ./hello_world.py it does not work, because it has no ConfigScope
The fewer dependencies the better
there are warnings about typechanges in the config that are not really helpful:
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/local/lib/python2.7/dist-packages/matplotlib/_pylab_helpers.py", line 86, in destroy_all
manager.destroy()
File "/usr/local/lib/python2.7/dist-packages/matplotlib/backends/backend_gtk3.py", line 427, in destroy
self.canvas.destroy()
AttributeError: FigureManagerGTK3Agg instance has no attribute 'canvas'
The term module
is used for pythons internal packaging system a lot and might lead to confusions. We should use a different term. Suggestions:
For those cases we should format the stacktrace ourselves and remove the noise
The current behaviour is to override assignments, which in this most basic case is fine:
@ConfigScope
def cfg():
a = [1, 2]
cfg(fixed={a:[7, 8, 9]})['a'] # returns [7, 8, 9]
But as soon as you start modifying your list it gets weird:
@ConfigScope
def cfg():
a = [1, 2]
a += [3, 4]
cfg(fixed={a:[7, 8, 9]})['a'] # returns [7, 8, 9, 3, 4]
Similar things happen with append
, insert
, extend
, *=
, pop
, remove
, del
, sort
, reverse
. This can be fixed by injecting an immutable list, which then simply ignores all the changes.
This removes the weirdness from before it also completely removes the possibility of adding something to a list afterwards. Also it seems a little inconsistent to me to have blocking dictionary that is open to changes apart from the forced ones, but have a list that is completely immutable. But for a lack of better ideas this is still the path I'll take.
Capture the console output of a program and write it to the database
The experiment will then have a mainfile entry like this
u'/home/greff/<ipython-input-1-762a1b9c0e46>'
There should be a way of manually setting the mainfile.
If a config element is changed by command line and its type changes we should issue a warning.
maybe like this:
@ex.config
def cfg():
dataset = {'path':'/', 'variant':'foo'}
pass
@ex.capture('dataset')
def load(path, variant):
pass
Because loading the sourcefile fails
It would be useful be able to restart the seeding at any point even while running an experiment.
right now only the mainfile is added to the DB entry. If modules are from different files they are ignored
provide an option to change loglevel on the commandline.
Maybe
-l LEVEL --logging LEVEL
where LEVEL can be either an int or the level string debug, info, warn, error, ...
adding/type-changing nested values leads to wrong coloring of parent
Allow to reuse code from a base-experiment with some configuration.
Have the experiment upon failure fire up a pdb post-mortem debugger.
This should only happen on automain trigger.
This should be deactivateable with a commandline option.
So far we support lazy strings (without quotes) only in direct assignments. E.g.: name=bob
. But it would be nice to also have that in lists and dicts:
a=[bob,lisa,hugo]
b={name:bob,age:23}
instead of
a=["bob","lisa","hugo"]
b={"name":"bob","age":23}
This would be a big gain because entering quotes on the commandline is painful.
Because there is no Run the captured functions won't be set up, so dependency injection does not work for them.
Implement the print_config command
Right now the Experiment
class is used for the set-up, and then a Run
is created for initialization and running. These two steps should be split.
main could maybe just be a regular command.
If so, we would have to define what is the default command, or do away with default commands altogether.
This should be decided soon, as it might break the API.
We should use GridFS to save big numpy arrays to the database. That would enable saving the weights of the network alongside the experiment.
Right now info
belongs to Run
but is accessed through Experiment
. This is not very clean. We could maybe use the dependency injected run to store info?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.