pattrn-project / pattrn Goto Github PK

initially, to support:

• CSV files (via XHR)
• JSON files (via XHR)
• pattrn API (see: #2)

common feature in visualizations of events over time (e.g. FA's own wherethedronesstrike.com)

• slider with configurable resolution
• ability to switch between different resolutions (e.g. year, quarter, month, week, day, etc.) within a visualization
• option to allow play button (starts progressing through timeline periods)
  • can be paused or stopped
  • option to loop (restart from beginning once finished playing timeline)
  • option to set play direction
  • option to allow automatic start of play in response to events (page loaded, etc.)
• option to play whole timeline or specific fragments in response to events (e.g. activating parts of a narrative, clicking on buttons, etc.)
• option to visually highlight subsets of fragments of timeline
• option to add external data (e.g. world events etc.)

Hey,
I saw the Forensic Architects talk at logan and I am interested in helping out. After skimming the codebase, I believe it could really benefit from using a build step / splitting it up into modules that get combined at build-time. This would accomplish a couple of things:

• Improve load-times by dramatically decreasing number of http requests needed to render the project
• Easier dependency management by using npm instead of CDN-hosted versions
• Ability to split the codebase into smaller, more testable modules (over time), improving separation of concerns and code readability
• Ability to explicitly set dependencies for parts of the library, making it easier for future contributors to grasp what is needed for each functionality
• Ability to write and run unit-tests for parts of the library

If there is interest, I could take a shot at setting up the boilerplate

accommodating events with arbitrary (within reason and implementing only what is useful to current projects) geographic extents:

• point + radius
• polygons
• proxies for administrative boundaries (countries/regions/cities/etc.) - e.g. centroids or capital city, etc.

this has been ported over from v1 and moved into its own module but the related DC chart is not being generated

Using:

• node v7.10.0
• npm 4.2.0

I'm getting the following error when I'm doing npm install:

$ npm install                                          

> [email protected] install /Users/edgar/sandboxes/pattrn/node_modules/contextify
> node-gyp rebuild

  CXX(target) Release/obj.target/contextify/src/contextify.o
../src/contextify.cc:131:56: warning: 'NewInstance' is deprecated [-Wdeprecated-declarations]
        Local<Object> wrapper = Nan::New(constructor)->NewInstance();
                                                       ^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3662:52: note: 'NewInstance' has been explicitly marked
      deprecated here
  V8_DEPRECATED("Use maybe version", Local<Object> NewInstance() const);
                                                   ^
../src/contextify.cc:150:16: error: no member named 'SetAccessCheckCallbacks' in 'v8::ObjectTemplate'
        otmpl->SetAccessCheckCallbacks(GlobalPropertyNamedAccessCheck,
        ~~~~~  ^
../src/contextify.cc:182:51: warning: 'GetRealNamedProperty' is deprecated [-Wdeprecated-declarations]
        Local<Value> rv = Nan::New(ctx->sandbox)->GetRealNamedProperty(property);
                                                  ^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3234:30: note: 'GetRealNamedProperty' has been explicitly
      marked deprecated here
                Local<Value> GetRealNamedProperty(Local<String> key));
                             ^
../src/contextify.cc:209:38: warning: 'GetRealNamedProperty' is deprecated [-Wdeprecated-declarations]
        if (!Nan::New(ctx->sandbox)->GetRealNamedProperty(property).IsEmpty() ||
                                     ^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3234:30: note: 'GetRealNamedProperty' has been explicitly
      marked deprecated here
                Local<Value> GetRealNamedProperty(Local<String> key));
                             ^
../src/contextify.cc:210:42: warning: 'GetRealNamedProperty' is deprecated [-Wdeprecated-declarations]
            !Nan::New(ctx->proxyGlobal)->GetRealNamedProperty(property).IsEmpty()) {
                                         ^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3234:30: note: 'GetRealNamedProperty' has been explicitly
      marked deprecated here
                Local<Value> GetRealNamedProperty(Local<String> key));
                             ^
4 warnings and 1 error generated.
make: *** [Release/obj.target/contextify/src/contextify.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:194:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Darwin 16.5.0
gyp ERR! command "/usr/local/Cellar/node/7.10.0/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/edgar/sandboxes/pattrn/node_modules/contextify
gyp ERR! node -v v7.10.0
gyp ERR! node-gyp -v v3.5.0
gyp ERR! not ok
[email protected] /Users/edgar/sandboxes/pattrn
└── (empty)

npm WARN [email protected] requires a peer of grunt@>=0.4.0 but none was installed.
npm ERR! Darwin 16.5.0
npm ERR! argv "/usr/local/Cellar/node/7.10.0/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v7.10.0
npm ERR! npm  v4.2.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs contextify
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls contextify
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/edgar/.npm/_logs/2017-05-06T01_17_50_554Z-debug.log

Are you guys using a different version of node/npm?

there are several areas of overlap in the v1 Pattrn code (through jQuery etc.) - these should be removed in order to make the frontend design fully independent from underlying data.

this will also enable different themes to be implemented and applied to specific Pattrn instances.

in Pattrn v2.0, the list of variables has been moved from a popup menu (Pattrn v1.0) to a side panel, as we now support an arbitrary (and potentially large) number of variables and filters rather than the 15 max of Pattrn v1.0. Moreover, as an arbitrary number of layer groups can be used (#30, pending fixing of several bugs related to this new feature), the list of variables available on a Pattrn instance can now be very long.

However, these long lists can be hard to navigate, especially as the order of layer groups, layers and variables is generated automatically from the order of variable types and variables defined in each layer group's metadata.

In Pattrn v1.0 variables were grouped by variable type, with a submenu of the main filter popup menu being used for each variable type. This grouping could be reproduced in Pattrn v2.0: perhaps combined with an accordion visual layout, leaving variables within a group hidden by default, except those with an active filter applied.

However, it is likely that in several contexts variables of a different type are actually analytically related, making a simple per-variable type grouping less useful.

A fully configurable menu layout should be implemented to improve Pattrn's UX, directly mapping the analytical aims of each instance to the layout of layer groups, layers and variables lists.

Arabic and Hebrew could be needed sooner than others, but this is mainly to make sure that current design efforts take RTL scripts into account

As work progresses on #5 - documentation on how to build the dist tree will be soon needed.

now that we have introduced layers and layer groups, the previous workflow for the loading of tree data files is not making efficient use of network requests.

as tree variables are now defined (as XHR-loaded resources) for each layer, in order to optimize loading of resources for tree data shared across layers, the metadata file should define a master list of XHR-loaded resources and only reference these from individual layers' variables metadata.

we can then XHR-load all the required data files, and use them where necessary.

• update metadata.json schema
  • add a section where XHR-loaded resources are listed
  • reference resources from the master list from within metadata of tree-type variables
• update platform.js code
  • retrieve all tree data files (only for variables X layers that actually contain data)
  • use preloaded tree data files when attaching tree charts to visualization

as part of diagnosing and cleaning up the data issues appearing with the recent layer groups code, we need to make sure that one and only one layer group can be active at any given time.

an exclusive toggle should work as first iteration of this.

(likely not needed for production, but useful for debugging issues with layer groups behind a dev environment flag)

the initial version of layer groups feature draws markers of each source dataset with a source dataset-specific colour; the same colour is used for markers of the same source dataset across all layer groups.

this makes sense from an UX point of view for the moment, but in development mode it is useful to instantly know if a marker being displayed on a map in a certain colour belongs to the source dataset corresponding to that colour within a given layer group.

although legacy design is not responsive, there is actually a basic issue with layout at 'desktop' screen sizes: in most cases there should be no reason for the app not to fit the viewport, whereas this is not the case as vertical scrollbars are present even when not really needed (this would be the case when lists of photo or video thumbnails are shown).

(moved to #43)

currently multiple selections are all treated as AND.

this should be configurable on a per-variable basis.

future iterations may include the ability to indicate whether to use AND or OR during selection, although this is way beyond the scope of this specific issue.

good UX affordances are needed to make navigating to and away from side panels (filters&event details) intuitive

v1 use a single events counter; when refactoring to add support for layer groups we wanted to have per-layer counters, although DC doesn't seem to handle these properly.

we need at least a per-layer group counter, displayed in the same place as before the layer group work

easy now that we use Jade for views, useful towards our eventual switch to generating navigation trees via code, and useful for a future themes system.

done in b20810f

• use libsass
• bundle local copy of relevant dc/bootstrap/leaflet/lightgallery CSS assets

although Pattrn's core focus is on spatial data, we often deal with datasets which are only partially geocoded - either because the location of an event is altogether unknown/unreported or because it has not been possible to reliably geocode the location from the available location data (place name, etc.), especially when place names are only available in English translitteration rather than in the original language.

it would be useful to nevertheless be able to handle these data points, off the map, for example as a dot whose size is proportional to the number of events it holds (or just as a simple dot marker with number of events in it).

this way we could still capture useful data associated to non-geocoded events (filtering, etc.) rather than having to reject these events on import of the dataset.

for datasets that take a while to load, some visual feedback (spinning wheel, etc.) would be very helpful to confirm to users that the platform is indeed loading.

this is particularly relevant for v1 instances with 'large' datasets as loading their data via Tabletop takes much longer than plain GeoJSON data in gzipped responses in v2.

The main website for this project currently gives a beautyfied 404 to the viewer
Nothing found. Sorry, the post you are looking for is not available. Maybe you want to perform a search?

data analysis tools to be exposed in an optional view

this is meant to enable admins to perform quick checks on data (e.g. navigate to coordinate, display data structure, toggle display of errors, etc.)

initial bug for the feature implemented in #38: combining multiple union filters when one of them is based on [0,n] possible values per row (e.g. weapon type: more than one can apply to a single event) and another on [0,1] values (non-overlapping geo regions) does not yeld the expected result. the bars on the filter for the dimension with [0,n] possible values displays a higher count than what is eventually selected.

i18n support for UI is self-contained; for data, this involves the whole pipeline (data cleanup, ingestion scripts, web dashboard when it will be developed, storage, API, etc.) so most of it will need to be handled separately, but let's focus on UI and display of data here.

via swagger.io or similar

refactor main loop around layer/layer groups data structure:

• retrieve list of layer groups and their layers (this includes layer and layer group metadata, dataset(s) and crossfilter object(s)) (see #31)
• process variables and data separately for each layer group: as one dataset and group of variables (by variable type) for intersection layer groups, as distinct datasets and groups of variables (by variable type) for union layer groups
• create menu items accordingly
• create and populate charts accordingly
• create and populate map layers and layer groups accordingly
• wire in D3 dispatches as needed

Currently, when using a tree chart that maps a data column to values in a (JSON) tree, the chart code will cause all events whose data (for the tree column) does not have any corresponding mid in the JSON tree to be silently discarded from the global crossfilter, as no match is effectively found.

As this scenario is likely to be quite common, it needs to be handled gracefully within the tree code.

This could also be partially mitigated during source data preparation, by checking that there are at least no mismatches between values that need to be used in the tree column and the available mids of the JSON tree.

Within the tree chart code, this could be handled by adding a (top-level?) tree node 'other' (or any other name as appropriate - this should be configurable), mapping all the variable levels not present in the JSON tree as mids to this node just after loading the JSON tree for a variable and before invoking the chart-making function for it.

pros: being per-chart, it doesn't need to touch the actual dataset
cons: the extra top-level node may not be desirable; although data rows needing to be put into this bucked should normally be as few as possible (otherwise the whole mismatch between data and JSON tree mids would make the whole chart and filter(s) analytically weak)

Alternatively, data rows with unknown values (aka values that are not present as mids in the JSON tree) should be ignored altogether within each tree chart instance, although this is not likely to be doable without actually moving the rows with unknown values to some specific 'bucket'.

currently NAs in tag type variables are converted to Unknown string literal to let crossfilter sort/group data correctly, but unknown unknowns are then removed from display (bar charts).

nevertheless, known unknowns should still be displayed: ideally the behaviour should be configurable on a per-variable basis.

this is due to min-height: 74vh; on .infowindow elements (event detail panels), which was likely arbitrarily chosen to fit some reasonably tall screen in the Pattrn v1 codebase.

we currently simply drop any non-valid rows (i.e. missing date/time or lat/lon, or otherwise containing invalid data) whilst processing the GeoJSON data being loaded.

part of the filtering is already happening on data ingestion (pg_loader or POST to API), and in the longer term this is where all invalid content should be made available for review, including all relevant error messages that could help cleaning up the rows affected.

however, in the meanwhile - and whilst we load full datasets via XHR - it would be useful to make the list of rejected rows available within the frontend itself.

given that this is a somewhat transitory arrangement, we could probably just create a global list for this, and output to console a note about this and how to access the list.

rather than treating layer groups as actual map (and data) layers, treating them as simple views should make things easier to manage data-wise and code-wise.

main drawback of this approach is that in general only one layer group can be active at any given time, so analytical contexts in which showing overlaps of two or more layer groups is needed would need a different strategy.

initially targeting:

• smartphones
• tablets
• desktop screens

(breakpoints TBD)

this is a side effect of moving away from the jumbo closures of the legacy code - needs reviewing so that it can be handled properly in Pattrn v2.0

It would be useful to make available some simple and clear documentation dedicated to helping potential contributors (people wishing to help with Pattrn code/UX/visual design) getting on board quickly and efficiently.

(in parallel, we can also start labelling GitHub issues that are suitable for new contributors as such)

needed in any case, but right now urgent to avoid making crossfilter's life miserable (error Uncaught RangeError: Maximum call stack size exceeded in Chrome)

module to navigate to lat/lon coordinates, in admin view (#6)

using gulp-cozy or require-dir or a similar strategy

We occasionally handle dataset that contain case variables being explicitly marked as (variously) uncertain, (variously) undefined, unreported, etc., besides being plainly empty.

This should be properly accounted for - the fact that absent data is different from uncertain or unreported is sometimes relevant (sometimes it isn't).

This applies to all the current variable types (booleans, integers, tags).

is this used at all (probably for interface menus), and if so - could we do without it (and therefore its dependency on jQuery)?

most of what is now done through jQuery can be done through d3 (selections and DOM manipulation). plain JS should suffice for the rest.
should make it easier to use React or other virtual DOM later on if that is the direction we wish/need to take.

when displaying bar charts, if values are very different between column with highest value and column(s) with lowest values, the latter are barely visible.

• they can hardly be seen
• it's hard do click on them to toggle data filters, especially on touch devices

a non-linear scale could be used, and the whole column for each bar could be made clickable (and the label area underneath as well, while on this).

As work progresses on #2 and #4, documentation will be needed on how to connect a Pattrn deployment to a Pattrn API backend or GeoJSON static file.