catalystcode / project-fortis-pipeline

Project Fortis is a data ingestion, analysis and visualization pipeline.

This repository is outdated and was migrated to project-fortis.

---

---

---

deploy fortis pipeline

Deploy your own Fortis pipeline to an azure subscription through a single click.

Pipeline Architecture

A fully containerized realtime spark pipeline powered off Kubernetes.

Background

Project Fortis is a data ingestion, analysis and visualization pipeline. The Fortis pipeline collects social media conversations and postings from the public web and darknet data sources.

Related Repositories

• Spark-based Pipeline: https://github.com/CatalystCode/project-fortis-spark
• Dashboard Interface: https://github.com/CatalystCode/project-fortis-interfaces
• Graphql Services: https://github.com/CatalystCode/project-fortis-services
• Kubernetes Helm Charts: https://github.com/CatalystCode/charts

Documentation

• Deploy to Azure Steps
• Deploy to Local Machine Steps
• Upgrading the Fortis Spark Pipeline
• Upgrading the Fortis Interface

Demo Videos

• Dashboard Walkthrough in Spanish

Deployment Prerequisites

• First and foremost, you'll need an existing azure subscription. You can create one for free here.
• Generate a Public / Private ssh key pair following these instructions. The contents from the generated MyKey.pub file will be used for the SSh Public Key field.
• You'll need an existing azure service principal. You can follow these instructions if you need to generate a new service principal. Your service principles Application ID will be used for the Service Principal App ID field, and the Authentication Key will be used for the Service Principal App Key.

Fortis Monitored Data Sources

• Public Web - Bing
• Reddit
• Twitter
• Facebook
• Instagram
• Radio Broadcasts
• ACLED

Site Types

• The site type selection drives which default topics, public sites and facebook pages are auto-generated for your site as part of the deployment process.
• Available site types
  • Humanitarian
  • Climate Change
  • Health

Post Deployment Instructions

• Grab a large cup of coffee as the deployment can take north of an hour to complete.

• Once the deployment has finished, click on the Manage your resources (highlighted below).

• Select the Tags tab in the Azure Portal (highlighted below), point your browser to the site at the FORTIS_ADMIN_INTERFACE_URL (also highlighted below).

• In the Fortis admin portal, you can now finalize the setup of your Fortis deployment; once you've completed all the admin configuration, your deployment is ready to be used.

project-fortis-pipeline's Issues

Navigating to "undefined" site shows valid dashboard

Navigate to
http://fortis-portal.52.167.224.142.nip.io/#/sites/
Click on Dashboard
http://fortis-portal.52.167.224.142.nip.io/#/site/undefined/

No PowerBI export

Customers are asking for export to PowerBI to build custom dashboards or to combine fortis data with other data sources.

Can't reset map to configured geofence

After zooming into the map, the only way to restore to the initial view is to refresh the page. Refreshing the page loses all other filter settings as well.

Would be great to have a way to reset the map to the configured area.

Chart Pop-Up Item Order not in sync with chart

Hover over timeline graph
Wait for pop up to come up.
Order of items in the pop up does not reflect order of lines on the chart, which makes it hard to relate the pop up to the chart. https://epicstuff.slack.com/files/U4VLP4QBS/F7UA4DQ79/image.png

Would be great if order of items matched the chart.

Admin site does not display configured event sources

Fortis V1 displayed lists of configured facebook and twitter accounts in great detail.

V2 only displays "Twitter" and "Facebook". No details about which accounts are configured on the admin page.

Makes it very hard to verify and troubleshoot for users that don't understand how to query the configuration from the database.

Verify item count and data quality

Bubbles on the map appear to display the count of items in a location. I believe, when you select a bubble, it shows the associated feed it's supposed to show the feed items of that bubble.

Sometimes it shows more items. See screenshot where the bubble has a count of 1, but 4 items are displayed: https://epicstuff.slack.com/files/U4VLP4QBS/F7T3VGMRR/image.png

Improve post-deployment site configuration experience

First run experience must be performable by a novice user without knowledge of low level packages like Cassandra.

Some ideas:

• Wizard driven config for sources
• Step-by-stp wizard for all required config steps (search terms, geo fence, sources)
• import cloned config and customize from existing config #178
• Search based definition of geo fence, i.e. type location name and system will look up geo fence CatalystCode/project-fortis@cfe7afb
• shared config library, i.e. new sites present a library of configs made available by other users #178
• curated configs made available from Microsoft or crowd sourcing #178

Deployment Tool does not display site URL or admin URL

Per docs at https://github.com/CatalystCode/project-fortis-pipeline#post-deployment-instructions user is expected to go into the Azure portal, navigate to non-intuitive fields resourcegroup->deployments->[deploymentname] and look up a field called FORTISSITEADMINURL.

1. in most of my attempts, this field is empty
2. step to retrieve the URL is hard to perform even for an IT admin familiar with Azure. Would be great to display that URL on the page that launched the deployment once the deployment finished successfully.

Site doesn't come up in IE

Site sits there with loading message. No errors in F12

If IE is not supported, then we should display an "unsupported browser" page.

Source selection Formatting in Edge has line breaks

Edge (top) breaks lines
Chrome (bottom) doesn't

No progress indicator or error messages when page starts

Page sits at "loading [sitename] watcher..." providing no indication that things are happening. It's confusing over slower connections because the user doesn't know that the site is making progress loading.
The page also does not display an error when something goes wrong and the page does not load, leaving the user wondering if loading every completes.

Setup only allows a single geofence to filter events

Customers may want finer grained configuration options, beyond a single rectangle to define the desired origin of events. Combined with lack of cloning a site, configuring a site for non-contiguous geographies is too hard.

Missing text in Top 5 Terms list

The Top 5 term list is not consistent in showing Top filter terms. Only terms 2 - 5 have text. Term 1 is missing the term name.

See: https://files.slack.com/files-tmb/T02L6HQ0H-F7SPR2W04-efcc5c8db1/image_480.png

Missing Admin Audit capabilities

There does not seem to be a capability to see audit logs of modifications.

Would be great for debugging and possible legal reasons if user account and originating IP address for config changes were recorded and audit logs were available.

Other Terms in term filter seems to be wrong

Other Terms in the Term Filter widget is checked, but none of the terms underneath is: https://files.slack.com/files-pri/T02L6HQ0H-F7TBFM5GV/image.png

Should Other Terms only be checked when terms below are selected?

Sentiment color doesn't match

On several occasions I had sentiment colors not match:

• Bubble on map doesn't match color in the item feed (https://epicstuff.slack.com/files/U4VLP4QBS/F7U3VV1C6/image.png)
• Item feed doesn't match pop up

(can't send repro steps since site currently doesn't load)

Translate Button only available when switching languages

Translate capability is great, but very hard to discover.

Consider a non-native speaker using the default configuration of a site. The translate capability is only revealed after switching the UI language.

Top Terms inconsistent

• Open site http://fortis-portal.52.167.224.142.nip.io/#/site/armedconflict
• select Last Week

Word Cloud shows muert as #2 most popular term with 495 mentions. muert does not show in the Top 5 terms, even though it has more mentions than the displayed term capturado with 113 mentions.

Blue sentiment color in item feed

Blue doesn't seem to be used as sentiment color anywhere else:

Inconsistent UI

• bring up site: http://fortis-portal.52.167.224.142.nip.io/#/site/armedconflict
• change filter to rafaga (gunfire)
• all Widgets update, including wordcloud showing different terms

word cloud does not include new filter term (rafaga). Might be by design, but it's confusing

Inconsistent UI: Locations and Terms on item details

Locations (left) and search terms (right) are both shown as blue pills, but only locations are highlighted in the text

Can't reset UI after changing timeline filter

Repro steps:

navigate to site.
change timeline filter to narrow down time filter

Clicking on Reset does not restore timeline to setting selected by the dropdown in the upper right corner.

Suggestion:

• Remove Reset Button,
• Rename to reflect what it does
• Move interval dropdown and timeline widget closer together since the former controls scope of the latter.

Credentials displayed in clear text

Credentials to Microsoft and 3rd party Services (facebook, twitter) are displayed in full in clear text on the admin site

To protect credential leaks, applications obfuscate tokens and credentials in the UI, some times displaying partial credentials, some don't display them at all.
(examples: GitHub, Azure Portal, Amazon, Google Cloud)

Timeline widget uses weird start dates

Timeline chart used Sat - Sat range for "Last Week" with seemingly arbitrary start and end times.
The first day of week should come from the users first day of week setting.
The weird start time may be due to timezone conversions.

Term Filters: language inconsistent

• open site
• change language to en
• type gunfire in filter list and wait for UI to change

Top 5 terms still has some, but not all Spanish items:

Language Change does not stick

• open http://fortis-portal.52.167.224.142.nip.io/#/site/armedconflict
• change language to en
• refresh page (ctrl-r)

Page is back in spanish

Geo Fence not labelled

Maps sometimes display a recognized location as an area on the map:

https://files.slack.com/files-pri/T02L6HQ0H-F7TAQSYH3/image.png

I can't tell where on the UI one finds a description or name of the particular area that's drawn on the map

VM SKU filter in ARM template is not consistent with nodeSelector in helm chart

ARM template allows all SKUs with local SSD, i.e. the full range of DS, L, GS, etc. VM sizes. CassaHelm chart only deploys Cassandra on a single VM SKU (L4s), see https://github.com/erikschlegel/charts/blob/spark-localssd/incubator/cassandra/templates/statefulset.yaml#L30 and https://github.com/erikschlegel/charts/blob/spark-localssd/incubator/cassandra/values.yaml#L10.

In clusters with VM size any other than L4S, the Azure deployment will finish successfully, i.e. the user thinks everthing is up and running, but fortis is not deployed because kubernetes is waiting with the Cassandra install until L4s nodes join the cluster.

Possible remedies:

• remove the nodeSelector for Cassandra from the helm chart
• restrict ARM template to L4S deployment only

Also, consider a Custom Script Extension as part of the template that will only finish successfully when fortis is up and running to avoid customer confusion when deployment reports success, but the deployment is up and running.

Item Pop up does not display images

Often, posts and articles include a picture that's very important to understand the item.

The image is not displayed. Neither is the link to the image clickable. A user has to copy the link, open another browser window and then paste the link to fully understand what an item was about.

Local Environment Health-Check Scripts & ReadMe

Create a health-check script for Windows & Linux that verifies all correct Fortis dependencies are properly installed.

Script should include the following criteria

• 1. java & javac health check for v1.8 or greater and OpenJDK version.
• 2. mvn check for v3.0 or greater
• 3. node & npm check for v5.0 and v6.0 respectively.
• 4. scala check for v2.0 or greater.
• 5. sbt check for v 0.13 or greater.
• 6. cassandra check for v3.0 or greater
• 7. spark path check and verify v2.7 or greater.
• 8. kubectl check .
• 9. helm validation.
• 10. kube config validate it exists and non-empty.
• 11. Redirect additional issues to Fortis Docs for guidance

UI doesn't load, bowser stuck at loading armedconfilict watcher...

Opening

http://fortis-portal.52.167.224.142.nip.io/#/site/armedconfilict

page stuck at:

loading armedconfilict watcher...

UI never comes up. Intermittent, but reproduces fairly reliably, almost half the time in Edge and Chrome. Happened on hotel wifi in NYC and Istanbul. F12 shows the UI loaded

Time axis time formatting not internationalized

Label on the time axis formatted for US am/pm time format. Since fortis is targeting international deployments, it would be better to format time according to local settings:

Blue area displaying geofence only in default zoom setting

Repro steps:

• Find a number bubble on the map where blue area bounds are displayed, as in https://files.slack.com/files-pri/T02L6HQ0H-F7TAQSYH3/image.png
• zoom in to the area, with plenty of space around the blue area

Blue area is not redrawn after zooming

Feed Item text doesn't show keyword

Snippet doesn't select the paragraph with the matched keyword from a longer text. A side effect is that the user can not get a translated text showing the keyword in context at all because in order to see the paragraph with the keyword, the user has to expand to the item details, but the item details cannot be translated.

https://epicstuff.slack.com/files/U4VLP4QBS/F7TBRT89K/image.png

Filters dropdown is confusing

Not sure what this dropdown does:
https://files.slack.com/files-pri/T02L6HQ0H-F7U34RUF8/image.png

I don't believe it does what it says, i.e. you don't search FOR a trusted source, term a or location.

I also don't believe the term "trusted source" is still used in the V2 UI

Filters input fails intermittently

• Pick a different filter in the filters input (rafaga, captura)
• either hit enter or tab out

Sometimes it works and sometimes it doesn't

Translate Button doesn't size properly

Translate button renders over text: https://epicstuff.slack.com/files/U4VLP4QBS/F7TBRT89K/image.png

No easy way to clone a site.

There does not seem to be an easy way to retrieve config settings for a deployed fortis site.

Would be greatly helpful to retrieve site settings and deploy another site with the same settings as a starting point for a new site.

Example: UNDP may want to deploy sites with the same event stream configurations for different geographies. It appears that today every site needs to be set up from scratch or it requires data export / import from the underlying database, which an casual user can probably not perform.

Timeline pop up missing values

Popup only displays one value, even though there are 10 lines in the graph.

Might be something specific to the chart widget, but it's confusing to the user.

Pop-Ups are not searchable

• Expand larger news item from the item feed (pick an RSS-based item).
• Press CTRL-F to search on the page

Not being able to search make it very hard to find keywords in a longer text, especially since keywords are not highlighted in the item text.

Language Change incomplete

• open http://fortis-portal.52.167.224.142.nip.io/#/site/armedconflict
• switch language to English

Top 1 search term language does not change in word cloud or Top 5 list. All others change

Time not formatted according to local settings

• Hover over timeline
• Wait for pop-up

Time displayed is formatted in ISO format. End users are not very familiar with ISO formatted times. Would be better to read time display format from local settings to also account for customary display in all regions around the world.

Facebook links don't show facebook content

Clicking on the link in the item feed or on the Read Original link for facebook posts comes out at a facebook page that the site is not available.

Maybe get the content from the API instead of redirecting?

Opening Translated Feed Item does not show translated text

Steps:

Translate a feed item with the Translate Button -> Item text is translated
Click on feed item to read the full text of the feed item -> text is displayed in original language

https://epicstuff.slack.com/files/U4VLP4QBS/F7U9CT6DD/image.png

No translate button on the Event Details pop-up

The Translate button on the feed widget is great.

However, to understand the text of a larger event text, like RSS feed item, users expand the feed item. Unfortunately the pop up with the full item text does not allow for translating the text.

Enable configuration of supportedLanguage via Azure deployment template

We want the ability to remove the hardcoded "en" supported language in the create-site call. This value should ideally come from the Azure deployment UI.

Suggestion: Color code Tags on item details

Similar experience looking at pages in Google or Bing cache where tags found in the text are highlighted in different colors.
You could color each tag (and even each location) with a different color and then highlight matched accordingly in the text

Navigating to partial URL shows valid looking results

Navigating to http://fortis-portal.52.167.224.142.nip.io/#/sites/

shows a site header, not an error or a message that the site doesn't exist.
Dashboard link brings up an undefined site.

No support for sharing dashboard views.

Dashboards do not allow easy sharing of current filter settings with a simple Share button on the UI or via copy and paste the URL with all the filter settings.

Users are used to being able to share a particular view from apps like Google Maps, Amazon, GitHub, etc.

Admin pages don't require authentication / authorization

Anybody that know the site URL scheme can get admin access site to any fortis site.