critocrito / sugarcube

Monoidal data processes.

Sugarcube

Synopsis

Sugarcube is a framework to fetch, transform and export data. Data processes are described using plugins, which are chained in sequence to model complex data processes.

It is a tool designed to support journalists, non-profits, academic researchers, human rights organisations and others with investigations using online, publicly-available sources (e.g.tweets, videos, public databases, websites, online databases).

Learn how to use Sugarcube on your own project.

This code is licensed under the GPL 3.

Documentation

All documentation can be found on the website.

• About Sugarcube
• Installation
• Tutorial

Examples

There are more examples and explanations on the website. Here is one to get you started.

sugarcube -p http_import,media_warc,media_screenshot,elastic_export \
          -c config.json \
          -Q http_url:'https://mwatana.org/en/airstrike-on-detention-center/'

This example will fetch and extract the contents and meta data of an online article, archive the website as a Web ARChive, take a screenshot of the website and store the data in an Elasticsearch database.

Data processes, like from the example above, can be codified in order to repeat them. Once a data process has been defined, Sugarcube allows to scale and automate it's operation.

Testimony

• Syrian Archive uses Sugarcube to archive video evidence of human rights violations in Syria. Further, Sugarcube is used to monitor human rights documentation that is taken down by social media companies. The systems and workflows developed with Syrian Archive are now being expanded to do similar work in Yemen, Sudan and other areas.

• Built using Sugarcube, the Data Scores investigation tool provided evidence and insights for research into how data analytics and data-driven "scoring" were being used in the public sector of the UK to make decisions. This research was conducted by the Data Justice Lab.

License

Sugarcube is licensed under the GPL 3.0.