wikimedia / research-datasets Goto Github PK

# archived - see https://gitlab.wikimedia.org/repos/research

# research repository

This repository serves multiple purposes
- library for sharing and reusing code for research at wmf. E.g. pipeline transformations for
    - graphs revision graph, diffs 
    - redirect resolution, link graph between articles
    - topic modelling
    - ...
- datasets. Pipelines that create datasets as output
- tooling for running orchestrated machine learning pipelines in the data engineering infrastructure
    - running airflow dags
    - gpu training on yarn cluster
    - custom workloads using skein 


Instead of splitting different functionality into different repos prematurely, this will be a mono repo for now. 

## Library

To use this package, add it as a requirement to your setup.py or pip install it into a given python evironment.

`pip install "wmf_research @ git+https://github.com/wikimedia/research-datasets.git@master"`


### Use interactively

todo. describe interactive ipython notebook 

## Datasets

### covid19

datasets about the covid19 pandemic that the wmf released publicly

### commons images

download images uploaded to wikipedia commons from swift and make the available on hdfs for machine learning use cases