asreview / automated-systematic-review-simulations

Simulations for the https://github.com/asreview/asreview

Automated Systematic Review - Simulation [DEPRECATED]

This project contains the code of the simulation study for the Automated Systematic Review project. It contains code to run batches of simulation runs in parallel using MPI.

We make use of the SURFSara HPC infrastructure. But with some modifications, the code can be run on other HPC facilities as well.

Some of the code in the repository is old an no longer maintained. Batch functionality has been (or will soon be) integrated into the core ASReview project. Other scripts will be copied to more suitable repositories.

Installation

The Automated Systematic Review project requires Python 3.6+. To run the code you also need an implementation of the MPI standard. The most well known standard is OpenMPI. This is not a python package and should be installed separately.

The simulation project itself can be directly installed with:

pip install --user git+https://github.com/asreview/automated-systematic-review-simulations

Dependencies are automatically installed.

Running a batch

To run a batch of simulations on 4 cores and 12 runs, use the following command:

mpirun -n 4 asreview batch ${DATA_SET} --state_file ${DIR}/results.json --n_runs 12

It will create 12 files in the ${DIR} directory, while running on 4 cores in parallel.

Related packages

• asreview-visualization Package for visualization of log files.

• asreview-hyperopt Package for optimizing ASReview hyperparameters.

automated-systematic-review-simulations's Issues

Informative error message - overwriting output file error

Describe the bug
When the results of a simulation are stored in a specific file and folder, it is not possible to overwrite the file when altering a part of the simulation command. Instead an error is returned.

When trying to run a simulation with code of Gerbrich's master thesis, I ran into the following problem. I had already run her original code a little while ago, which is why a file was already created in the directed folder with a specific name. When then trying to alter the code a bit (only adding the n_papers argument), an error occurred:
TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'.

This error was fixed by simply altering the state_file command such that the results will be saved in a different file or folder.

To Reproduce
A while ago, ran the code:
asreview simulate ../../datasets/sim_datasets/nudging.csv --config_file config/one/BCTD/nb_max_double_tfidf-nudging.ini --state_file simoutput/one/BCTD/nudging/results.h5 --init_seed 42

Then today tried to run:
asreview simulate ../../datasets/sim_datasets/nudging.csv --config_file config/one/BCTD/nb_max_double_tfidf-nudging.ini --state_file simoutput/one/BCTD/nudging/results.h5 --init_seed 42 --n_papers 200

This resulted in the error provided in the screenshot below.

To fix, simply change the folder in which the results are stored:
asreview simulate ../../datasets/sim_datasets/nudging.csv --config_file config/one/BCTD/nb_max_double_tfidf-nudging.ini --state_file simoutput/one/BCTD/nudging2/results.h5 --init_seed 42 --n_papers 200

The difference can be found in the --state_file part

Screenshots
This resulted in the following output:

Version information

• Windows Version 10.0.18363 Build 18363
• asreview-simulation-0.1

Additional context
Might be something to discuss in the upcoming docathon?