ctuning / ck-artifact-evaluation

Collective Knowledge repository to support artifact evaluation and reproducibility initiatives:

News: we have moved all main AE pages to our MLCommons CM repository maintained by the MLCommons taskforce on education and reproducibility.

Data license: Code license:

This is a Collective Knowledge repository with methodology, templates and workflows to reproduce results from published articles (artifact evaluation). See Artifact Evaluation evaluation website for more details.

Description

Install the Collective Knowledge framework (CK) on Linux, Windows or MacOS as described here.

Pull this repository:

ck pull repo:ck-artifact-evaluation

Note that the dependencies on ck-web and other CK repositories will be automatically resolved by the CK framework!

 $ ck start web
 $ Open web page: http://localhost:3344/?template=artifact-evaluation&page=index

That's all!

Note, that it is also possible to create a static website for http://cTuning.org/ae as follows:

 $ cd `script/compile-website-adapt-workshop`
 $ ./create_website.sh

or on Windows:

 $ ck find script/compile-website-adapt-workshop
 $ cd /D {above directory}
 $ create_website.bat

CK will create all static html pages in the "tmp-website" sub-directory that you can upload to your web server.

You can generate a list of artifacts with their ACM badges from the HotCRP JSON file as follows:

 $ ck process_hotcrp ae --json_file={downloaded JSON file with decisions from HotCRP}

You can also validate your artifact XML before adding it to the ACM DL as follows:

Contributions

Just send a pull request and contact Grigori Fursin to upload your updates to cTuning.org/ae .

Public discussions

• Artifact Evaluation google group

Artifacts in the CK format

We promote collaborative and reproducible research where artifacts and workflows are shared along with papers in a common format to let the community quickly reuse them and thus accelerate open science!

You can find all papers with artifacts shared in the CK format here.

Please check the ACM proceedings of our 1st ACM ReQUEST tournament with a live scoreboard and all artifacts and workflows shared in a common CK format.

You can also check this reusable and customizable artifact from CGO'17 with automatic cross-platform software installation and web-based experimental dashboard powered by the CK framework:

• GitHub CK repo
• Paper with artifact appendix
• PDF snapshot of the interactive CK dashboard

Our publications

• "Community-driven Reviewing and Validation of Publications"
• "Collective Knowledge: organizing research projects as a database ofreusable components and portable workflows with common APIs"