thvasilo / uncertain-trees-reproducible

Online random forests with prediction uncertainty

Reproducibility repository for Online Regression Forests

Using this repository you should be able to reproduce all the experiments we performed for our JMLR paper on online regression forests with uncertainty.

Follow the instructions to prepare you environment and data. The file reproduce-output.sh contains the commands to re-create the most important tables and figures in the paper.

Instructions

The repository uses submodules to keep track of the different repositories needed to run the algorithms, so ensure you clone using the --recursive option, i.e. git clone --recursive https://github.com/thvasilo/uncertain-trees-reproducible.git

Installing the dependencies

uncertain-trees-experiments and scikit-garden

There are a few Python libraries needed to run the project, so we recommend creating a virtual environment to avoid messing up your default environment. We have used the Anaconda Python distribution to make things easier.

We've made some small modifications to the original scikit-garden library, so we need to install it from the included submodule rather than the PyPI repository.

conda env create -f rf-pred.yml  # Installs the base dependencies as a new virtual env
source activate rf-pred
pip install -e ./scikit-garden  # Install the customized scikit-garden repo.

MOA

We recommend using the pre-built binaries under binaries. The only requirement is Java 8. We've tested with the Oracle JDK, OpenJDK seems to cause issues with the results.

Alternatively you can build the MOA distribution using Maven by running mvn package -DskipTests in moa/moa.

Obtaining the data

The stationary data are included with the repository under the data/small-mid directory. The large airlines data are compressed under data/airlines. To decompress them, cd into data/airlines and run:

for FILE in *.tar.gz; do tar -zxf ${FILE}; done

To re-create the Friedman data run the generate_friedman_data.sh script.

Re-creating the files

It's also possible to re-create the files using the scripts we've included in the data/airlines directory.

You just need to run in succession:

./get_data.sh
./create_splits.sh

These two scripts will pull the original data, transform to csv, apply the pre-processing steps, and create the 700k, 2M and 5M splits in arff format using Weka.

Running the experiments

After you've prepared the environment and data, to re-run the experiments from the paper we can use the example commands in reproduce-output.sh. We recommend running the experiments selectively and not simply running the script, because the runtime for the airlines experiments is very long. The experiments on the small-scale data should not take very long however.

NOTE: Due to the random nature of the algorithms the exact results will be slightly different from those reported in the paper, unfortunately we didn't keep track of all the random seeds used in our experiments. The overall performance of the algorithms should not change significantly however.

Troubleshooting

Ensure you did git clone --recursive https://github.com/thvasilo/uncertain-trees-reproducible.git. Please file an issue if you run into any problems.

Citing

If you use this work please cite our JMLR paper:

@article{JMLR:v20:19-006,
  author  = {Theodore Vasiloudis and Gianmarco De Francisci Morales and Henrik Bostr{{\"o}}m},
  title   = {Quantifying Uncertainty in Online Regression Forests},
  journal = {Journal of Machine Learning Research},
  year    = {2019},
  volume  = {20},
  number  = {155},
  pages   = {1-35},
  url     = {http://jmlr.org/papers/v20/19-006.html}
}