critocrito / sugarcube Goto Github PK

The schema has to work both with SQLite and PostgreSQL. The schema also has to support annotations and annotation templates.

The elastic_export plugin increments new and existing metrics for itself. It would be interesting to break those down by the source of the unit, and set appropriate counters.

• <plugin>.[new|existing]

Just getting up and running here. I thought I'd mention the following test failures with the latest code:

  1) maybeDate
       parses string in German format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 83152e9af3b47d086c; Counterexample: "2014-05-17T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

  2) maybeDate
       parses string in US format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 8d481cf3d6ea705b29; Counterexample: "2014-12-06T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

  3) maybeDate
       parses string in ISO8601 format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 89d64c78afbe295031; Counterexample: "2014-11-28T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

  4) maybeDate
       parses string in little endian hyphened format:
     Error: Failed after 1 tests and 0 shrinks. rngState: 82bfdc39e08271a564; Counterexample: "2014-10-02T00:00:00.000Z"; 
      at /media/kratib/data/src/meedan/sugarcube/node_modules/jsverify/lib/jsverify.js:360:15
      at Object.map (node_modules/jsverify/lib/functor.js:35:12)
      at checkThrow (node_modules/jsverify/lib/jsverify.js:354:30)
      at Context.<anonymous> (node_modules/jsverify/lib/jsverify.js:400:14)

Thanks!

Develop a schema for Sugarcube data that works for SQLite as well as PostgreSQL.

Version 7 of Elasticsearch deprecated some things, especially related to the way mappings are handled. Not sure if the current implementation works on the new version. This relates to #3 to use the new @elastic/elasticsearch NodeJS library.

The current implementation is not efficient enough and breaks down when checking a lot of tweets at once (more than ~150.000).

InfluxDB is the only database currently that can persist Sugarcube instrumentation. This is operationally complex and prevents the use of metrics for local-first deploys of Sugarcube.

While query import from sheets can use a default type, moving the query to a different sheet fails, since it doesn't respect the default type.

The make-docs.js script currently only generates documentation for the plugins. Add support for instruments as well.

There are a whole bunch of plugins that are either broken (e.g. instagram) or it is unclear if they are still working.

Plugins that are known to be broken:

• instagram_feed
• facebook_api_user
• facebook_api_page
• facebook_api_feed

Plugins that haven't been tested in a long time:

• aqicn_station
• google_reverse_images_files
• google_images
• google_search
• guardian_search
• http_wget
• regex_match
• sec_search

Sugarcube supports two data stores for persistency, Elasticsearch and MongoDB. I want to be able to store data in Postgresql and SQLite as well.

@sugarcube/utils/install.js file is set to download an old version of Tika (1.22) from a broken link and fails. Should change to latest Tika version.

For Syrian Archive I configured mappings for Sugarcube data for Elasticsearch. They are more refined and better tested. The automatic mappings in the elastic_export plugin could use some update based on those.

Currently failing stats are tracked as they are discovered in statsd and reports are sent as failing stats. This obfuscates takedown reports for end users. It would be great to have a single stat that gives the number and percentage of takedown content and to produce dedicated reports of those stats per month.

Not sure yet if I want to convert the whole codebase to Typescript, but providing typings for core would be great.

Sugarcube sets the _sc_source field, which is as a default set to the plugin name. This can lead to situations where I have two tweets, one has the source set to twitter_feed and the other to twitter_tweets. But both units are actually tweets. Add a new field, e.g. _sc_data_source, that can identify the data source across different plugins.

The elasticsearch client currently in use is deprecated. It is replaced by the @lastic/elasticsearch package. They provide a migration guide which has a good overview of the changes.