Use
from empiricalci import empiricalci
...
empiricalci.saveOverall('average', avg)
:microscope: Empirical CLI
Home Page: https://empiricalci.com
License: MIT License
1) buildImage should reject if there is an error:
Error: done() called multiple times
at Suite.<anonymous> (test/index.js:149:3)
at Object.<anonymous> (test/index.js:148:1)
at Array.forEach (native)
at node.js:972:3
2) buildImage should reject if there is an error:
Error: done() called multiple times
at Suite.<anonymous> (test/index.js:149:3)
at Object.<anonymous> (test/index.js:148:1)
at Array.forEach (native)
at node.js:972:3
README.md
should have information directed to the final user on how to setup and run emp.DEVELOPMENT.md
_id
of the experiment to the evaluator/standalone container along with API credentials via environment variablesSave the json
files in the workspace and list the results in the YML file:
results: /workspace/results.json
$HOME/empirical
emp configure
will allow to change the empirical directory/workspace
directory for the experiment--save
After updating all 3 branches of https://github.com/empiricalci/hello-world to use empiricalci.saveOverall(metric, value)
and updating the seeds to use the latest commits, make sure that the tests are passing.
Sometimes the solver terminates before the solver starts.
ci-worker-1 | 2016-03-25T01:25:57.447228208Z RUN EXPERIMENTS: [ { evaluator: 'empirical-bot/my-evaluator:EJJoSOppg' } ]
ci-worker-1 | 2016-03-25T01:25:57.447378981Z RUNNING EXPERIMENT: { evaluator: 'empirical-bot/my-evaluator:EJJoSOppg' }
ci-worker-1 | 2016-03-25T01:25:57.724781683Z RUN LINKED { image: 'empirical-bot/my-solver:VylkzKppg',
ci-worker-1 | 2016-03-25T01:25:57.724846986Z name: 'solver-56f493a55cfe6d0300d2427e' } { image: 'empirical-bot/my-evaluator:EJJoSOppg',
ci-worker-1 | 2016-03-25T01:25:57.725054188Z name: 'evaluator-56f493a55cfe6d0300d2427e',
ci-worker-1 | 2016-03-25T01:25:57.725165582Z env:
ci-worker-1 | 2016-03-25T01:25:57.725203244Z [ 'EXPERIMENT_ID=56f493a55cfe6d0300d2427e',
ci-worker-1 | 2016-03-25T01:25:57.725331876Z 'EMPIRICAL_API_URL=http://qa.empiricalci.com/api/x' ] }
ci-worker-1 | 2016-03-25T01:25:57.725446260Z solver-56f493a55cfe6d0300d2427e: Running container
ci-worker-1 | 2016-03-25T01:25:57.753841674Z solver-56f493a55cfe6d0300d2427e: Container created
ci-worker-1 | 2016-03-25T01:25:57.753895885Z evaluator-56f493a55cfe6d0300d2427e: Running container
ci-worker-1 | 2016-03-25T01:25:57.759062220Z evaluator-56f493a55cfe6d0300d2427e: Finished running
ci-worker-1 | 2016-03-25T01:25:57.759158691Z DATA: null
ci-worker-1 | 2016-03-25T01:25:57.759680221Z CONTAINER: null
ci-worker-1 | 2016-03-25T01:25:57.762660314Z stopping container 147e9d963746b717c94073cc1ae84532b3f75d94b49533713097f270d8120c31
ci-worker-1 | 2016-03-25T01:25:57.981389252Z Empirical: Solver is now running
If there's an error during the build, it's not being caught anywhere.
Because channels with /
are not valid channel names.
Remove the mockups that are in there
It's not checking out the correct version. Check the logs. (LOGS.log)
Currently emp is being distributed via a docker image, however this might be too heavy specially for users with node.
Be able to execute multiple runs of a protocol in parallel.
Separate into:
BUILD:
{"stream":"Step 1 : FROM python:2.7\n"}
{"stream":" ---\u003e f9a9ac5dcfb8\n"}{"stream":"Step 2 : RUN pip install numpy\n"}
undefined:2
{"stream":"Step 2 : RUN pip install numpy\n"}
^
SyntaxError: Unexpected token {
at Object.parse (native)
at IncomingMessage.<anonymous> (/Users/empirical/workspace/emp/lib/build-image.js:16:30)
at emitOne (events.js:77:13)
at IncomingMessage.emit (events.js:169:7)
at readableAddChunk (_stream_readable.js:153:18)
at IncomingMessage.Readable.push (_stream_readable.js:111:10)
at HTTPParser.parserOnBody (_http_common.js:127:22)
at Socket.socketOnData (_http_client.js:322:20)
at emitOne (events.js:77:13)
at Socket.emit (events.js:169:7)
Looks like 2 json lines are being merged into one, which makes it invalid json
While developing, you may want to explore your container and do multiple tests in the same container without having to build a new image and launch a container every time that you make changes. In this scenario having the option to run experiments interactively is desired.
emp run -i protocol /path/to/code
emp listen
should launch the listen.js
scriptemp run experiment-name /path/
should run a experiment from the given pathemp run empiricalci/hello-world/sample/x35cg_d3
should fetch the experiment information from empiricalci.com and run itemp
shows usage instructions & versionemp versions
returns versionIt's currently failing due to the use of readlink
to convert from relative to absolute path. Maybe use this:
#!/bin/bash
realpath() {
[[ $1 = /* ]] && echo "$1" || echo "$PWD/${1#./}"
}
realpath "$0"
from here
emp run
will now run the experiment locally only and generate a report of the experimentemp push
should publish the report to empiricalci.comThat way this repo can run all its test without depending on the Server.
The experiments that are being currently used to run the tests are too slow. Use dummy experiments instead, to optimize the tests.
Refactor experiments to build the images related to one experiment first and then execute the experiment. Then move to the next experiment.
In the future this will also allow to parallelize experiments later
The user is assumed to have pushed the commit that is going to be tested to GitHub first. This will trigger a webhook that will create a new version
on the server without any experiments.
emp run experiment /path/to/code --save user/project
version
associated with the current sha
.version
is found , the experiment will be created and associated with that versionversion
is not found, the experiment will not run and it will throw the following error: "Couldn't find theversion
with sha 99fb56a on project user/proejct. Did you pushed this commit to GitHub?"NOTE: Depending on feedback, future work will be saving to empiricalci.com experiments not associated with GitHub.
Some questions to get started:
Replace "experiments" for "protocols" to be in line with the rest of the system's nomenclature
When using the docker distribution, make sure that the install path is not overwritten when mounting files and directories from the host.
Windows compatibility using node.
From here. The simplest way is to just expose the Docker socket to your CI container, by bind-mounting it with the -v
flag.
Simply put, when you start your CI container (Jenkins or other), instead of hacking something together with Docker-in-Docker, start it with:
docker run -v /var/run/docker.sock:/var/run/docker.sock ...
Now this container will have access to the Docker socket, and will therefore be able to start containers. Except that instead of starting "child" containers, it will start "sibling" containers.
If your CI makes use of the Docker binary in scripts, you can include it in your CI image, or bind-mount it from the host was well. Example:
docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/bin/docker \
-ti ubuntu
Only print stack trace on development mode
Define Installation and update workflow for:
Don't require keys for public experiments. So that public repos can be cloned by other users:
Try to get keys
If (unauthorized)
try using https url without keys
If it fails: return unauthorized
if it succeeds continue with the process
There are a few scenarios:
Currently the whole data directory is being mounted to /data
on the experiment container. The user then needs to refer to the files using its hash which is not very user friendly.
We could instead mount the files individually and refer to them by the key name.
For example a data file with the following config:
"myfile.csv": {
"url": "http://example.com/myfilesdsds.csv",
"hash": "sdfd39asaea334353fdfh4sfsd3432353"
}
would be mounted from the host path /home/empirical/data/sdfd39asaea334353fdfh4sfsd3432353
to the container path /data/myfile.csv
GPU_ENABLED
environment variable to experiment container- [ ] Package a directory and get hash emp data tar directory
emp data hash directory
emp data hash file
emp data get file
emp data get directory
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.