REMARKs are self-contained and complete projects, whose content here should be executable by anyone with a suitably configured computer or using nbreproduce.
Types of content include (see below for elaboration):
- Explorations
- Use the Econ-ARK/HARK toolkit to demonstrate some set of modeling ideas
- Replications
- Attempts to replicate important results of published papers written using other tools
- Reproductions
- Code that reproduces ALL of the results of some paper that was originally written using the toolkit
Each project lives in its own repository. To make a new REMARK, you can start with a skeleton REMARK-starter-example and add to it, or from an example of a complete project using the toolkit, BufferStockTheory, whose content (code, text, figures, etc) you can replace with your own.
REMARKs should adhere to the REMARK Standard.
The REMARK catalog and Econ-ARK website configuration will be maintained by Editors.
Editorial guidelines are here.
A catalog of all REMARKs is available under the REMARK tab at econ-ark.org.
The ballpark is a place for the set of papers that we would be delighted to have replicated in the Econ-ARK.
In cases where the replication's author is satisfied that the main results of the paper have been successfully replicated, we expect to approve pull requests for new REMARKs with minimal review, as long as they satsify the criteria described in the Standard.
We also expect to approve with little review cases where the author has a clear explanation of discrepancies between the paper's published results and the results in the replication attempt.
We are NOT intending this resource to be viewed as an endorsement of the replication; instead, it is a place for itb to be posted publicly for other people to see and form judgments on. Nevertheless, pull requests for attempted replications that are unsuccessful for unknown reasons will require a bit more attention from the Econ-ARK staff, which may include contacting the original author(s) to see if they can explain the discrepancies, or may include consulting with experts in the particular area in question.
Replication archives should contain two kinds of content (along with explanatory material):
- Code that attempts to replicate key results of the paper
- A Jupyter notebook that presents at least a minimal set of examples of the use of the code.
This material will all be stored in a directory with some short pithy name (a bibtex citekey might make a good directory name).
Code archives should contain:
- All information required to get the replication code to run
- Including a
requirements.txtfile explaining the software requirements
- Including a
- An indication of how long it takes to run the
reproduce.shscript- One some particular machine whose characteristics should be described
Jupyter notebook(s) should:
- Explain their own content ("This notebook uses the associated replication archive to demonstrate three central results from the paper of [original author]: The consumption function and the distribution of wealth)
- Be usable for someone wanting to explore the replication interactively (so, no cell should take more than a minute or two to execute on a laptop)
The key difference with the contents of the DemARK repo is that REMARKs are allowed to rely on the existence of local files and subdirectories (figures; data) at a predictable filepath relative to the location of the root.
Command Line Interface cli.py
cli.py is an automated tool that facilitates:
- cloning of REMARK repositories
- linting (detection of missing files from a given REMARK)
- building conda environments/docker images
- uses
conda/repo2dockerunder the hood
- uses
- executing
reproduce.shscripts within the built environments.
All artifacts generated by cli.py are stored in a newly created _REMARK folder.
- Once you clone a REMARK you'll be able to find its contents inside of
_REMARK/repos/… - Once you build/execute a REMARK you'll be able to find a corresponding log
file from that process inside of
_REMARK/logs/…
cli.py has built-in parallelization specified by the -J flag for many actions.
- python 3.9 or newer.
- contents
requirements.txt
Clone/Pull
pulling REMARKs (these are populated in the _REMARKS folder)
python cli.py pull --all # git clone all REMARKS
python cli.py pull {remark_name} # git clone one or more REMARK(s)Lint
Shows what files are missing from given REMARK(s). The linter uses the file-tree print out from STANDARD.md and compares it to the current files found in the currently cloned REMARK(s).
python cli.py lint --all # detect missing files from all REMARKs
python cli.py lint {remark_name} # detect missing files from one or more REMARK(s)Build
Building conda environments and/or docker images.
python cli.py build conda --all # build conda environments for all REMARKs (stored as a `condaenv` folder inside the cloned REMARK repo)
python cli.py build docker --all # build docker images for all REMARKs (stored as a `condaenv` folder inside the cloned REMARK repo)
python cli.py build conda {remark_name} # build conda environments for one or more REMARK(s)
python cli.py build docker {remark_name} # build docker image(s) for one or more REMARK(s)The primary difference between conda and docker for builds are that docker will be more flexible for multilanguage REMARKs. It leverages
repo2docker (same tool that mybinder uses) to create docker images from repositories.
Execute
Automated execution within built conda environments/docker containers.
python cli.py execute conda --all # execute reproduce.sh via conda for all REMARKs
python cli.py execute docker --all # execute reproduce.sh via docker for all REMARKs
python cli.py execute conda {remark_name} # execute reproduce.sh via conda for one or more REMARK(s)
python cli.py execute docker {remark_name} # execute reproduce.sh via docker for one or more REMARK(s)Both the build and execute subcommands have an optional --jobs argument to specify the number of jobs to run in parallel when building/executing.
Logs/Summarize
python cli.py logs # view most recent logs for all previous building/executing commandsClean/Remove
python cli.py clean conda --all # remove all built conda environments
python cli.py clean docker --all # remove all build docker images
python cli.py clean conda {remark_name} # remove conda environment(s) from specified REMARK(s)
python cli.py clean docker {remark_name} # remove docker images built from specified REMARK(s)
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent