pattrn-project / pattrn Goto Github PK
View Code? Open in Web Editor NEWData-driven, participatory fact-mapping
Home Page: http://pattrn.co
License: GNU Affero General Public License v3.0
Data-driven, participatory fact-mapping
Home Page: http://pattrn.co
License: GNU Affero General Public License v3.0
initially, to support:
common feature in visualizations of events over time (e.g. FA's own wherethedronesstrike.com)
Hey,
I saw the Forensic Architects talk at logan and I am interested in helping out. After skimming the codebase, I believe it could really benefit from using a build step / splitting it up into modules that get combined at build-time. This would accomplish a couple of things:
If there is interest, I could take a shot at setting up the boilerplate
accommodating events with arbitrary (within reason and implementing only what is useful to current projects) geographic extents:
this has been ported over from v1 and moved into its own module but the related DC chart is not being generated
Using:
I'm getting the following error when I'm doing npm install
:
$ npm install
> [email protected] install /Users/edgar/sandboxes/pattrn/node_modules/contextify
> node-gyp rebuild
CXX(target) Release/obj.target/contextify/src/contextify.o
../src/contextify.cc:131:56: warning: 'NewInstance' is deprecated [-Wdeprecated-declarations]
Local<Object> wrapper = Nan::New(constructor)->NewInstance();
^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3662:52: note: 'NewInstance' has been explicitly marked
deprecated here
V8_DEPRECATED("Use maybe version", Local<Object> NewInstance() const);
^
../src/contextify.cc:150:16: error: no member named 'SetAccessCheckCallbacks' in 'v8::ObjectTemplate'
otmpl->SetAccessCheckCallbacks(GlobalPropertyNamedAccessCheck,
~~~~~ ^
../src/contextify.cc:182:51: warning: 'GetRealNamedProperty' is deprecated [-Wdeprecated-declarations]
Local<Value> rv = Nan::New(ctx->sandbox)->GetRealNamedProperty(property);
^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3234:30: note: 'GetRealNamedProperty' has been explicitly
marked deprecated here
Local<Value> GetRealNamedProperty(Local<String> key));
^
../src/contextify.cc:209:38: warning: 'GetRealNamedProperty' is deprecated [-Wdeprecated-declarations]
if (!Nan::New(ctx->sandbox)->GetRealNamedProperty(property).IsEmpty() ||
^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3234:30: note: 'GetRealNamedProperty' has been explicitly
marked deprecated here
Local<Value> GetRealNamedProperty(Local<String> key));
^
../src/contextify.cc:210:42: warning: 'GetRealNamedProperty' is deprecated [-Wdeprecated-declarations]
!Nan::New(ctx->proxyGlobal)->GetRealNamedProperty(property).IsEmpty()) {
^
/Users/edgar/.node-gyp/7.10.0/include/node/v8.h:3234:30: note: 'GetRealNamedProperty' has been explicitly
marked deprecated here
Local<Value> GetRealNamedProperty(Local<String> key));
^
4 warnings and 1 error generated.
make: *** [Release/obj.target/contextify/src/contextify.o] Error 1
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:276:23)
gyp ERR! stack at emitTwo (events.js:106:13)
gyp ERR! stack at ChildProcess.emit (events.js:194:7)
gyp ERR! stack at Process.ChildProcess._handle.onexit (internal/child_process.js:215:12)
gyp ERR! System Darwin 16.5.0
gyp ERR! command "/usr/local/Cellar/node/7.10.0/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /Users/edgar/sandboxes/pattrn/node_modules/contextify
gyp ERR! node -v v7.10.0
gyp ERR! node-gyp -v v3.5.0
gyp ERR! not ok
[email protected] /Users/edgar/sandboxes/pattrn
└── (empty)
npm WARN [email protected] requires a peer of grunt@>=0.4.0 but none was installed.
npm ERR! Darwin 16.5.0
npm ERR! argv "/usr/local/Cellar/node/7.10.0/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v7.10.0
npm ERR! npm v4.2.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the contextify package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp rebuild
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs contextify
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls contextify
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /Users/edgar/.npm/_logs/2017-05-06T01_17_50_554Z-debug.log
Are you guys using a different version of node/npm?
there are several areas of overlap in the v1 Pattrn code (through jQuery etc.) - these should be removed in order to make the frontend design fully independent from underlying data.
this will also enable different themes to be implemented and applied to specific Pattrn instances.
in Pattrn v2.0, the list of variables has been moved from a popup menu (Pattrn v1.0) to a side panel, as we now support an arbitrary (and potentially large) number of variables and filters rather than the 15 max of Pattrn v1.0. Moreover, as an arbitrary number of layer groups can be used (#30, pending fixing of several bugs related to this new feature), the list of variables available on a Pattrn instance can now be very long.
However, these long lists can be hard to navigate, especially as the order of layer groups, layers and variables is generated automatically from the order of variable types and variables defined in each layer group's metadata.
In Pattrn v1.0 variables were grouped by variable type, with a submenu of the main filter popup menu being used for each variable type. This grouping could be reproduced in Pattrn v2.0: perhaps combined with an accordion visual layout, leaving variables within a group hidden by default, except those with an active filter applied.
However, it is likely that in several contexts variables of a different type are actually analytically related, making a simple per-variable type grouping less useful.
A fully configurable menu layout should be implemented to improve Pattrn's UX, directly mapping the analytical aims of each instance to the layout of layer groups, layers and variables lists.
Arabic and Hebrew could be needed sooner than others, but this is mainly to make sure that current design efforts take RTL scripts into account
As work progresses on #5 - documentation on how to build the dist tree will be soon needed.
now that we have introduced layers and layer groups, the previous workflow for the loading of tree data files is not making efficient use of network requests.
as tree variables are now defined (as XHR-loaded resources) for each layer, in order to optimize loading of resources for tree data shared across layers, the metadata file should define a master list of XHR-loaded resources and only reference these from individual layers' variables metadata.
we can then XHR-load all the required data files, and use them where necessary.
as part of diagnosing and cleaning up the data issues appearing with the recent layer groups code, we need to make sure that one and only one layer group can be active at any given time.
an exclusive toggle should work as first iteration of this.
(likely not needed for production, but useful for debugging issues with layer groups behind a dev environment flag)
the initial version of layer groups feature draws markers of each source dataset with a source dataset-specific colour; the same colour is used for markers of the same source dataset across all layer groups.
this makes sense from an UX point of view for the moment, but in development mode it is useful to instantly know if a marker being displayed on a map in a certain colour belongs to the source dataset corresponding to that colour within a given layer group.
although legacy design is not responsive, there is actually a basic issue with layout at 'desktop' screen sizes: in most cases there should be no reason for the app not to fit the viewport, whereas this is not the case as vertical scrollbars are present even when not really needed (this would be the case when lists of photo or video thumbnails are shown).
(moved to #43)
currently multiple selections are all treated as AND.
this should be configurable on a per-variable basis.
future iterations may include the ability to indicate whether to use AND or OR during selection, although this is way beyond the scope of this specific issue.
good UX affordances are needed to make navigating to and away from side panels (filters&event details) intuitive
v1 use a single events counter; when refactoring to add support for layer groups we wanted to have per-layer counters, although DC doesn't seem to handle these properly.
we need at least a per-layer group counter, displayed in the same place as before the layer group work
easy now that we use Jade for views, useful towards our eventual switch to generating navigation trees via code, and useful for a future themes system.
done in b20810f
although Pattrn's core focus is on spatial data, we often deal with datasets which are only partially geocoded - either because the location of an event is altogether unknown/unreported or because it has not been possible to reliably geocode the location from the available location data (place name, etc.), especially when place names are only available in English translitteration rather than in the original language.
it would be useful to nevertheless be able to handle these data points, off the map, for example as a dot whose size is proportional to the number of events it holds (or just as a simple dot marker with number of events in it).
this way we could still capture useful data associated to non-geocoded events (filtering, etc.) rather than having to reject these events on import of the dataset.
for datasets that take a while to load, some visual feedback (spinning wheel, etc.) would be very helpful to confirm to users that the platform is indeed loading.
this is particularly relevant for v1 instances with 'large' datasets as loading their data via Tabletop takes much longer than plain GeoJSON data in gzipped responses in v2.
The main website for this project currently gives a beautyfied 404 to the viewer
Nothing found. Sorry, the post you are looking for is not available. Maybe you want to perform a search?
data analysis tools to be exposed in an optional view
this is meant to enable admins to perform quick checks on data (e.g. navigate to coordinate, display data structure, toggle display of errors, etc.)
initial bug for the feature implemented in #38: combining multiple union filters when one of them is based on [0,n] possible values per row (e.g. weapon type: more than one can apply to a single event) and another on [0,1] values (non-overlapping geo regions) does not yeld the expected result. the bars on the filter for the dimension with [0,n] possible values displays a higher count than what is eventually selected.
i18n support for UI is self-contained; for data, this involves the whole pipeline (data cleanup, ingestion scripts, web dashboard when it will be developed, storage, API, etc.) so most of it will need to be handled separately, but let's focus on UI and display of data here.
via swagger.io or similar
refactor main loop around layer/layer groups data structure:
Currently, when using a tree chart that maps a data column to values in a (JSON) tree, the chart code will cause all events whose data (for the tree column) does not have any corresponding mid
in the JSON tree to be silently discarded from the global crossfilter, as no match is effectively found.
As this scenario is likely to be quite common, it needs to be handled gracefully within the tree code.
This could also be partially mitigated during source data preparation, by checking that there are at least no mismatches between values that need to be used in the tree column and the available mid
s of the JSON tree.
Within the tree chart code, this could be handled by adding a (top-level?) tree node 'other' (or any other name as appropriate - this should be configurable), mapping all the variable levels not present in the JSON tree as mid
s to this node just after loading the JSON tree for a variable and before invoking the chart-making function for it.
pros: being per-chart, it doesn't need to touch the actual dataset
cons: the extra top-level node may not be desirable; although data rows needing to be put into this bucked should normally be as few as possible (otherwise the whole mismatch between data and JSON tree mid
s would make the whole chart and filter(s) analytically weak)
Alternatively, data rows with unknown values (aka values that are not present as mid
s in the JSON tree) should be ignored altogether within each tree chart instance, although this is not likely to be doable without actually moving the rows with unknown values to some specific 'bucket'.
currently NAs in tag type variables are converted to Unknown
string literal to let crossfilter sort/group data correctly, but unknown unknowns are then removed from display (bar charts).
nevertheless, known unknowns should still be displayed: ideally the behaviour should be configurable on a per-variable basis.
this is due to min-height: 74vh;
on .infowindow
elements (event detail panels), which was likely arbitrarily chosen to fit some reasonably tall screen in the Pattrn v1 codebase.
we currently simply drop any non-valid rows (i.e. missing date/time or lat/lon, or otherwise containing invalid data) whilst processing the GeoJSON data being loaded.
part of the filtering is already happening on data ingestion (pg_loader or POST to API), and in the longer term this is where all invalid content should be made available for review, including all relevant error messages that could help cleaning up the rows affected.
however, in the meanwhile - and whilst we load full datasets via XHR - it would be useful to make the list of rejected rows available within the frontend itself.
given that this is a somewhat transitory arrangement, we could probably just create a global list for this, and output to console a note about this and how to access the list.
rather than treating layer groups as actual map (and data) layers, treating them as simple views should make things easier to manage data-wise and code-wise.
main drawback of this approach is that in general only one layer group can be active at any given time, so analytical contexts in which showing overlaps of two or more layer groups is needed would need a different strategy.
initially targeting:
(breakpoints TBD)
this is a side effect of moving away from the jumbo closures of the legacy code - needs reviewing so that it can be handled properly in Pattrn v2.0
It would be useful to make available some simple and clear documentation dedicated to helping potential contributors (people wishing to help with Pattrn code/UX/visual design) getting on board quickly and efficiently.
(in parallel, we can also start labelling GitHub issues that are suitable for new contributors as such)
needed in any case, but right now urgent to avoid making crossfilter's life miserable (error Uncaught RangeError: Maximum call stack size exceeded
in Chrome)
module to navigate to lat/lon coordinates, in admin view (#6)
using gulp-cozy
or require-dir
or a similar strategy
We occasionally handle dataset that contain case variables being explicitly marked as (variously) uncertain, (variously) undefined, unreported, etc., besides being plainly empty.
This should be properly accounted for - the fact that absent data is different from uncertain or unreported is sometimes relevant (sometimes it isn't).
This applies to all the current variable types (booleans, integers, tags).
is this used at all (probably for interface menus), and if so - could we do without it (and therefore its dependency on jQuery)?
most of what is now done through jQuery can be done through d3 (selections and DOM manipulation). plain JS should suffice for the rest.
should make it easier to use React or other virtual DOM later on if that is the direction we wish/need to take.
when displaying bar charts, if values are very different between column with highest value and column(s) with lowest values, the latter are barely visible.
a non-linear scale could be used, and the whole column for each bar could be made clickable (and the label area underneath as well, while on this).
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.