critocrito / sugarcube

Monoidal data processes.

Sugarcube

Synopsis

Sugarcube is a framework to fetch, transform and export data. Data processes are described using plugins, which are chained in sequence to model complex data processes.

It is a tool designed to support journalists, non-profits, academic researchers, human rights organisations and others with investigations using online, publicly-available sources (e.g.tweets, videos, public databases, websites, online databases).

Learn how to use Sugarcube on your own project.

This code is licensed under the GPL 3.

Documentation

All documentation can be found on the website.

• About Sugarcube
• Installation
• Tutorial

Examples

There are more examples and explanations on the website. Here is one to get you started.

sugarcube -p http_import,media_warc,media_screenshot,elastic_export \
          -c config.json \
          -Q http_url:'https://mwatana.org/en/airstrike-on-detention-center/'

This example will fetch and extract the contents and meta data of an online article, archive the website as a Web ARChive, take a screenshot of the website and store the data in an Elasticsearch database.

Data processes, like from the example above, can be codified in order to repeat them. Once a data process has been defined, Sugarcube allows to scale and automate it's operation.

Testimony

• Syrian Archive uses Sugarcube to archive video evidence of human rights violations in Syria. Further, Sugarcube is used to monitor human rights documentation that is taken down by social media companies. The systems and workflows developed with Syrian Archive are now being expanded to do similar work in Yemen, Sudan and other areas.

• Built using Sugarcube, the Data Scores investigation tool provided evidence and insights for research into how data analytics and data-driven "scoring" were being used in the public sector of the UK to make decisions. This research was conducted by the Data Justice Lab.

License

Sugarcube is licensed under the GPL 3.0.

sugarcube's Issues

Write documentation for plugin-sql

Provide a migration process from MongoDB to SQL

Support Elasticsearch 6 and 7

Version 7 of Elasticsearch deprecated some things, especially related to the way mappings are handled. Not sure if the current implementation works on the new version. This relates to #3 to use the new @elastic/elasticsearch NodeJS library.

Distinguish beteen source plugin and data source type

Sugarcube sets the _sc_source field, which is as a default set to the plugin name. This can lead to situations where I have two tweets, one has the source set to twitter_feed and the other to twitter_tweets. But both units are actually tweets. Add a new field, e.g. _sc_data_source, that can identify the data source across different plugins.

Import and export queries to SQLite

Improve performance of youtube video failing plugin

Import and export units to SQLite

Use the offical Elasticsearch client

The elasticsearch client currently in use is deprecated. It is replaced by the @lastic/elasticsearch package. They provide a migration guide which has a good overview of the changes.

Failing tests

Just getting up and running here. I thought I'd mention the following test failures with the latest code:

  1) maybeDate
       parses string in German format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 83152e9af3b47d086c; Counterexample: "2014-05-17T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

  2) maybeDate
       parses string in US format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 8d481cf3d6ea705b29; Counterexample: "2014-12-06T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

  3) maybeDate
       parses string in ISO8601 format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 89d64c78afbe295031; Counterexample: "2014-11-28T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

  4) maybeDate
       parses string in little endian hyphened format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 82bfdc39e08271a564; Counterexample: "2014-10-02T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

Thanks!

Provide a migration process between SQL and Elasticsearch

Store metrics in SQL

InfluxDB is the only database currently that can persist Sugarcube instrumentation. This is operationally complex and prevents the use of metrics for local-first deploys of Sugarcube.

Instrument new and existing units by source

The elastic_export plugin increments new and existing metrics for itself. It would be interesting to break those down by the source of the unit, and set appropriate counters.

• <plugin>.[new|existing]

Define a SQL schema for Queries

The schema has to work both with SQLite and PostgreSQL. The schema also has to support annotations and annotation templates.

Provide Typescript typings

Not sure yet if I want to convert the whole codebase to Typescript, but providing typings for core would be great.

Define a SQL schema for Sugarcube units

Develop a schema for Sugarcube data that works for SQLite as well as PostgreSQL.

Change Tika version on install

@sugarcube/utils/install.js file is set to download an old version of Tika (1.22) from a broken link and fails. Should change to latest Tika version.

Update automatic mappings for Elasticsearch

For Syrian Archive I configured mappings for Sugarcube data for Elasticsearch. They are more refined and better tested. The automatic mappings in the elastic_export plugin could use some update based on those.

Googlesheets query_merge should respect the default query type

While query import from sheets can use a default type, moving the query to a different sheet fails, since it doesn't respect the default type.

Store Sugarcube data in a SQL database

Sugarcube supports two data stores for persistency, Elasticsearch and MongoDB. I want to be able to store data in Postgresql and SQLite as well.

Import and export queries to PostgreSQL

Improve reporting on Takedown Content

Currently failing stats are tracked as they are discovered in statsd and reports are sent as failing stats. This obfuscates takedown reports for end users. It would be great to have a single stat that gives the number and percentage of takedown content and to produce dedicated reports of those stats per month.

Deprecate broken plugins, move others to sugarcube-contrib.

There are a whole bunch of plugins that are either broken (e.g. instagram) or it is unclear if they are still working.

Plugins that are known to be broken:

• instagram_feed
• facebook_api_user
• facebook_api_page
• facebook_api_feed

Plugins that haven't been tested in a long time:

• aqicn_station
• google_reverse_images_files
• google_images
• google_search
• guardian_search
• http_wget
• regex_match
• sec_search

Generate documentation of instruments

The make-docs.js script currently only generates documentation for the plugins. Add support for instruments as well.

Import and export units to PostgreSQL

Add plugin to check Twitter tweets for takedown

The current implementation is not efficient enough and breaks down when checking a lot of tweets at once (more than ~150.000).