isabella232 / crowdtruth-core

CrowdTruth framework for crowdsourcing ground truth for training & evaluation of AI systems

This library processes crowdsourcing results from Amazon Mechanical Turk and CrowdFlower following the CrowdTruth methodology. A full description of the metrics is available in this paper. For more information see http://crowdtruth.org.

If you use this software in your research, please consider citing:

@article{CrowdTruth2,
  author    = {Anca Dumitrache and Oana Inel and Lora Aroyo and Benjamin Timmermans and Chris Welty},
  title     = {CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement},
  year      = {2018},
  url       = {https://arxiv.org/abs/1808.06080},
}

Useful links:

• Data collected with CrowdTruth
• Papers that use CrowdTruth
• Previous version CrowdTruth v.1.0

Installation

To install the stable version from PyPI, install pip for your OS, then install package using:

pip install crowdtruth

To install the latest version from source, download the library and install it using:

python setup.py install

Tutorial

The following tutorial is a collection of slides, exercises and Jupyter notebooks that explains what is the CrowdTruth methodology, and how to use it in practice. If you are already familiar with CrowdTruth, you can skip straight to the guide on how to run this library.

Introduction to CrowdTruth

• Slides

Task Design & Building CrowdTruth Annotation Vectors

• Slides
• Hands-on Exercises

Data Processing & CrowdTruth Metrics

• Slides
• Getting Started with the CrowdTruth Library
• Hands-on Exercises