allenai / quoref-leaderboard-example

Submission of XLNet baseline to the Quoref leaderboard

XLNet QA Baseline for Quoref

This repository shows an example submission to the Quoref leaderboard. The model is XLNet based QA model implemented by HuggingFace in pytorch-transformers. You can find a description of the Quoref dataset and the baseline models in our EMNLP 2019 paper.

The main script is run_model.sh which downloads the XLNet QA model trained on Quoref from S3 and predicts from /quoref/nolabels.json, which is the location at which the leaderboard mounts the input file. It runs two different Beaker experiments for development and test sets, and in both experiments, the input file is mounted at the same location. Note that /quoref/ is mounted as a read-only file system. If you need to write temporary files, please do so in /tmp/.

Follow the instructions below to create a Docker container and publish it as a Beaker image, which can then be submitted as a leaderboard entry.

Requirements

• Docker
• Beaker

Submitting to the leaderboard

Follow the these steps in the repository root directory:

1. Create a docker image:

    docker build -t xlnet_quoref_model .
   

2. Upload an image to beaker:

    beaker image create -n my-quoref-model-image xlnet_quoref_model:latest
   

   NOTE: The image name (e.g., my-quoref-model-image) should be unique in beaker under your account.

3. Make a submission:

   • Follow the steps on the Quoref leaderboard to gain access to making submissions.
   • On the new submission form, make sure to enter your image name (e.g., my-quoref-model-image).