allegro / turnilo

Business intelligence, data exploration and visualization web application for Druid, formerly known as Swiv and Pivot

Turnilo

Turnilo is a business intelligence, data exploration and visualization web application for Apache Druid. Turnilo is a fork of Pivot which is currently available under commercial licence only. This repository was forked from the stalled repository Swiv with the latest version of Pivot under Apache license.

Motivation

Druid is heavily used as business intelligence platform at Allegro. In order to gain wide adoption of non-technical users, Druid requires simple yet powerful user interface. In Allegro we have decided that we are going to continue Pivot development as an open source software, this is how Turnilo emerged.

Manifesto

• High usability for non-technical users over sophisticated but rarely used features.
• Focus on interactive data exploration over static predefined dashboards.
• Self-describing reports for users without deep domain expertise.
• Outstanding integration with Druid over support for other data sources like SQL databases.
• Focus on data visualizations over Druid cluster or data ingestion management.
• Data cubes configuration as a code over UI editor backed by non-versioned database.
• Stateless over stateful server-side architecture.
• Support for most recent versions of standards compliant browsers.

Features

• Intuitive, drag and drop, gorgeous user interface to visualize Druid datasets.
• Fully dedicated to low latency Druid Timeseries, TopN and GroupBy queries.
• Unified view for historical and real-time data.
• Blazingly fast.

Try it!

You can try an online demo with example datasets (Covid-19 and Wikipedia) at https://turnilo.app.

Join us!

Feel free to ask on GitHub Discussions or join the chat on Slack.

Pre-requisites

• Node.js - 14.x or 16.x version

⚠️ Do not use yarn command for dependency management and project build, use npm instead. With npm builds are reproducible (thanks to package-lock.json) and even faster than with yarn.

Usage

Install Turnilo distribution using npm.

npm install -g turnilo

Start off by running Turnilo with example datasets and open http://localhost:9090/.

turnilo run-examples

Use connect-druid command to connect to the existing Druid broker. Turnilo will automatically introspect your Druid broker and figure out available datasets.

turnilo connect-druid http[s]://druid-broker-hostname[:port]

Documentation

Learn how to configure and customize Turnilo: https://allegro.github.io/turnilo/

Development

Install project dependencies.

npm install

Build the project.

npm run build

Run project

Run example datasets.

npm run start:examples

Connect to the existing Druid broker.

npm run start -- connect-druid http[s]://druid-broker-hostname[:port]

Connect to the existing Druid broker using your config file.

npm run start -- run-config path/to/config.yml

Run project in developer mode

Every change in frontend code would recompile project and reload page.

Run example datasets.

npm run start:dev:examples

Connect to the existing Druid broker.

npm run start:dev -- connect-druid http[s]://druid-broker-hostname[:port]

Connect to the existing Druid broker using your config file.

npm run start:dev -- run-config path/to/config.yml

Testing

Unit tests

Run all unit tests.

npm run test

Or run tests separately for common, client and server modules.

npm run test:common
npm run test:client
npm run test:server

End to end tests

Run all e2e tests. It will start Turnilo with example datasets in background.

npm run e2e

End to end tests development

Run server

npm start:dev:examples

Run cypress interactive tools for e2e testing

npm run e2e:dev

Debugging

Server module

In WebStorm\IntelliJ open "Run/Debug Configurations", click "Add New Configuration". Next choose "Node.JS", set "JavaScript file" to "./bin/turnilo" and "Application parameters" to "--examples".

You can find more infrmation here

Generating documentation locally

Prerequisites

• Ruby
• Bundler

Go to the docs folder and:

1. Install bundle install or update bundle update dependencies
2. Run bundle exec jekyll serve --livereload
3. Open http://localhost:4000/

License

Turnilo is published under Apache License 2.0.

turnilo's Issues

Disable sourceListRefreshInterval and sourceReintrospectInterval for local development

Source refreshing and re-introspection should be disabled for local development (and --examples run) to avoid unnecessary logs on console. I recommend to change defaults in the code.

Corresponding configuration properties:

• sourceListScan
• sourceListRefreshOnLoad
• sourceListRefreshInterval
• sourceReintrospectOnLoad
• sourceReintrospectInterval

Add to documentation more details about refreshRule, e.g for realtime data cubes it should be set to "realtime" not "query" to avoid unnecessary query to Druid cluster. For "batch" data cubes it should be set to "query".

use react-transition-group，not react-addons-css-transition-group！

react-addons-css-transition-group is deprecated by facebook，use react-transition-group instead it！

Update to the latest plywood and compatible immutable-class and chronoshift packages

Get rid of swiv-playwood (version 0.12.17) - the library is available at npm package but there is no sources under yahoo/swiv project. The latest compatible version of implydata/plywood is 0.12.12. I do not have any clue what is a difference between 0.12.12 and 0.12.17.

My proposal is to migrate project to the latest official implydata/plywood (0.17.x or never) and update immutable-class and chronoshift packages to the compatible versions. It should also resolve #9.

Limit number of splits to avoid DDOS on Druid Cluster

Develop unfinished feature for limiting splits in data cube. Unlimited splits could kill Druid cluster.

See:

turnilo/src/client/views/cube-view/cube-view.tsx

Line 50 in 47b8b64

maxSplits?: number;

Allow histogram parameters tuning for quantile measures

Current implementation of plywood does not allow tuning of parameters for approximate histogram in quantile expressions.

It should be possible to use following expression:
$main.quantile($a_histogram,0.98,'resolution=100,numBuckets=11,lowerLimit=0,upperLimit=1000')

This issue should be resolved with #10 but needs to be verified.

Define tslint rules and enforce linting in build

• Review and define tslint rules
• Optionally define prettier rules
• Enable linting and make it's positive result a requirement in build

Health endpoint does not check anything

Health endpoint should check that broker is ready to be queried (e.g after restart).

Check proposal (GET request to the broker): /druid/broker/v1/loadstatus

Healthy response:

• HTTP/1.1 200 OK
• Content-Type: application/json
• Content: {"inventoryInitialized": true}

Health endpoint location should be defined in the configuration file (with default pointing existing location).

Place for data cube technical details (e.g number of dimensions/measures)

Place for data cube technical details:

• number of dimensions/measures
• retention
• granularity

Having clause and filterable metrics

It would be lovely if we could add a feature similar to Having clause. For example I would like to filter metric values only for people who bought at least 10 items. Or to see values for people who spent money in range (range filtering).

From the UI perspective it would be nice to add filter option for metrics (numerical values). Metrics should be filterable.

Can‘t drag and drop dimension in Microsoft's edge browser

Can‘t drag and drop dimension to split(or filter) in Microsoft's edge browser!

Create new logo

It would be nice to have new logo crafted for Turnilo only. Logo should be prepared in SVG format, files in other formats should be removed from resources/logo. New logo should be used to generate all required favicons under assets directory and the favicons generation should be automated.

Export to CSV is broken for Polish locales

In the exported CSV file field separator is always "," (comma). Unfortunately Excel with polish (and german AFAIK) locales expects ";" as a field separator, import with default settings is broken. User is able to choose separator during import but it requires some Excel ninja skills.

• configured globally in the configuration file (in the customization section)
• documented

Check also file encoding, perhaps it should be also configured.

Alternatively file could be exported as TSV, tab is recognised as field separator in Excel as well.

adding support for latest druid stable 0.12.0

Is is currently working with latest druid version and making use of latest druid enhancements in aggregration ?

Adding filter to overflow menu does not work

Adding filter or split that cannot be accommodated in filter or split tile and thus is allocated to overflow menu does not work.

In case of filter it does not present a filter selector and filter is left in hang state - it cannot be removed or selected.
In case of split it throws error in Console.

Authentication module for enabling limited access

Does it have authentication for enabling limited access ?

Auto-refresh option or refresh button does not work for real-time ingestion

Turnilo URLs and payload for /mkurl should be stable even if internal data model or plywood expressions are changing

Turnilo URLs and payload for /mkurl should be stable as public API. Turnilo internal changes should be fully encapsulated and must not break bookmarked links or mkurl payloads.

• New data model for URL encoded state
• Independent of Plywood expressions if feasible
• Version 3 of URL encoder
• Separate class for legacy URL encoder (for version 1 & 2)
• Separate class for new URL encoder (version 3)
• Support for versioned payload in mkurl
• Updated mkurl to the new encoding scheme
• URL model shared with mkurl

Move from numeral.js to numbro

Numeral.js is now a dead project with plenty nasty bugs. Numbro is a promising fork from numeral.js with similar API.

Cannot create split by clicking on dimension name

Clicking dimension name in the dimensions tile and clicking split icon does not create split when there is no split in cube view. None of the icons work, neither [ ] nor + icon.

On the other hand when there already is a split selected clicking dimension name and selecting split ([ ]) or add split (+) icon correctly replaces or adds another split. Dragging a dimension into split tile also works as expected.

Ability to hide side panels and make main visualization panel full screen.

On smaller screens I would like hide side panels, and make main visualization bigger.
This Hide/Show feature could be also applied to filter/split area.

Line chart's Legend tab is broken when replacing an existing split

Opening the following link with Turnilo started in wikipedia example mode presents an hourly line chart filtered for 7 "Country ISOs" and split by "Country ISO" with Legend tile on the right.

When dragging "Country Name" dimension to replace "Country ISO" split it breaks the Legend view which now shows "undefined" entries.

Add "percent of parent" and "percent of root" transformations for measures

It should be possible to add measures which have additional, special transformations configured.

Percent of parent should present split values as percent of a value in the parent split.
Percent of parent should present split values as percent of a total value.

Transformations should be possible to configure with a "transform" attribute on a measure with allowed values of "percent-of-parent" and "percent-of-root".

- name: percent
  formula: $main.count()
  transform: percent-of-parent

Dimensions and measures groups

As a Turnilo user I want to organise related dimensions and measures together. E.g UTM related dimensions (UTM Source, UTM Campaign, UTM Term, ...) should be presented under UTM group.

• Related fields should be presented under single tree branch with name as separate leafs.
• Grouping should be defined explicitly in the configuration file (additional level of aggregation under dimensions and measures elements).
• Dimension or measure name must be unique in the whole dataCube definition (not unique in the group).
• When visualization is open directly using url (e.g: from mkurl) and nested dimension or measure is used the group with this dimension or measure must be expanded.

Existing configuration structure:

dimensions:
    - name: dim1
      title: Dim 1
      formula: ...

    - name: dim2
      title: Dim 2
      formula: ...

    - name: dim3
      title: Dim 3
      formula: ...

measures:
    - name: measure1
      title: Measure 1
      formula: ...

    - name: measure2
      title: Measure 2
      formula: ...

    - name: measure3
      title: Measure 3
      formula: ...

Proposed configuration structure:

dimensions:
    - name: dim1
      title: Dim 1
      formula: ...

    - name: dim2
      title: Dim 2
      formula: ...

   - name: groupA
      type: group
      title: Group A
      dimensions:
        - name: dimA1
           title: Dim A 1
           formula: ...
  
        - name: dimA2
           title: Dim A 2
           formula: ...

    - name: dim3
      title: Dim 3
      formula: ...

measures:
    - name: measure1
      title: Measure 1
      formula: ...

    - name: measure2
      title: Measure 2
      formula: ...

    - name: groupA
      type: group
      title: Group A
      measures:
        - name: dimA1
           title: Dim A 1
           formula: ...
  
        - name: dimA2
           title: Dim A 2
           formula: ...

    - name: measure3
      title: Measure 3
      formula: ...

Default type for dimensions elements: "field" or "dimension".
Default type for measures elements: "field" or "measure".
Type for group: "group".

Disable Geo visualisation

Geo visualisation is not implemented so it should not be displayed as an option.

Server module tests are not run

Some of the routes in server import SETTINGS_MANAGER from config module which in turn tries to parse command line options.

When running mocha tests for these routes required options are not present on the command line and help message is printed instead of running tests.

It should be possible to pass settings mock in tests without trying to parse command line options.

Rich data cubes description on main page

In order to understand the meaning of data cubes presented on the main page
As an analyst (or any business user)
I want to have rich description of data cubes with hyperlinks to the external documentation.

• Description should be defined in the configuration file (already exists)
• Description defined as Markdown or plain text
• Description should be also available on the data cube page itself (with strong visual indicator activated on click)
• Without tooltip in the left menu (there is already native tooltip only)
• Search in the left menu (search by data cube titles only)
• Better visual indicator for hiding left menu (better than Turnilo logo)
• Data cubes as list (one column) on main page
• Search for data cubes on main page (titles and descriptions are searchable)
• Long description available under additional action like "see more ..."
• There is no need for "data cubes" header on main page

Avoid duplicated configuration for webpack

Common webpack configuration should be shared between development and production setup.
See: https://webpack.js.org/guides/production/

Could not define limit for the split in table visualization

Open turnilo and try to change limit for "Comment" split. "Infinity" is always set.

Get rid of commited typings

Types for TypeScript are commited under /typings folder. Types should be declared as npm dependencies if exist. Folder /typings should be used only for types not available as npm packages.

Data Grid visualization for splits on 2 dimensions and 1 measure

New Grid like visualization, one split for X axis, second for Y axis, single measure values in grid cells.
When column or row is selected, selected value is proposed as a new filter (include or exclude). When filter is applied, grid is transformed into single row or column. When cell is selected, selected value is proposed as new filters on both dimensions (X and Y, include or exlude). When filter is applied, grid is transformed into single cell.

Time comparison on table, chart and bar visualisations

• It should allow to compare the same time period in the past
• Only for single split based on time
• Selected on time split modal
• Compared to the same period of time in the past
• On the chart and bar views second time series should be presented as a dashed line and the percentage on the tooltip
• On table view two additional columns should be presented (metric for the compared time period and percentage)
• Single query to the Druid cluster for both time ranges if feasible in plywood
• Comparison must be bookmarkable, it also means that mkurl must handle this kind of visualisation properly.

remove qajax，use fetch（or axios）instead

qajax is now a dead project，use fetch or axios instead it！

Missing time selector for continuous dimension

"Specific" tab for continuous dimension does not allow to specify time, only date.
The only way to select specify fragment of the day is to use line chart and then select the time range from the chart.

App fails to start if installed with yarn

When turnulo npm package is installed with yarn, it fails to start

This is caused by different algorithm for version hoisting

immutable-class package made API-breaking change in patch release, and removed static method isInstanceOf somewhere around version 0.6.20 (though 0 major version allows to do everything)

turnilo explicitly depends on 0.6.9 version of immutable class, but swiv-plywood depends on 0.6.x, which causing issues when package is installed with yarn. It installes 0.6.42 for plywood and 0.6.9 for turnilo itself, but swiv-plywood 0.12.17 can't work with immutable-class 0.6.41

BTW I was not able to find a repo of swiv-plywood, npm page references original repository

Relative time filter labels

When current or previous relative time filter is selected its label should be set to "Current/Latest/Previous day/week/month/quarter/year".

Display Druid query

For debug purposes it would be nice to display Druid query (queries) used for generating current Turnilo view. Right now, the queries are shown only on console in verbose mode.

TODO: Define UI.

Remove unused code

Step 1: mark unused code
Step 2: discuss what should be remove
Step 3: remove unused code

Hide "collections" on the main screen

Collections presented on the main screen is a not implemented feature (or partially implemented). Hide "collections" section on the main screen to avoid bug reports from Turnilo users.

Configurable theme support

Default Turnilo orange theme is aligned to Allegro L&F guidelines. It would be nice to configure Turnilo with different colour scheme.

• Theme should be configured as single property with theme name in the configuration file.
• All SCSS variables for given theme should be defined in separate file.
• SCSS variables for theme must be documented.
• Allegro orange theme must be default one
• Swiv/PIvot blue theme should be available out of the box second theme.

when remove split，also remove the legend from pinboard at the same time

if line-chart visualization have two split（such as： channel and comment length），the first split(e.g channel) is acted as Legend, the second split(e.g comment length) acted as x-axis。
first step： remove the second split(e.g comment length)。
next step： remove the first split(e.g channel)，then a error appeared!
so,
when remove the second split(e.g comment length), should also remove the legend from pinboard at the same time。

Change Look&Feel using Allegro colour pallete

IntelliJ does not recognise TypeScript compiler configuration for common and server module

TypeScript compiler configuration files should be aligned to application modules (common, client, server). Currently IntelliJ uses only default tsconfig.json which is valid only for client module.

"View Raw Data" - Cannot read property 'height' of null

Exception is thrown for "View Raw Data" action:

Uncaught TypeError: Cannot read property 'height' of null
    at RawDataModal.Array.concat.RawDataModal.renderRows (raw-data-modal.tsx:226)
    at RawDataModal.Array.concat.RawDataModal.render (raw-data-modal.tsx:296)
    at finishClassComponent (react-dom.development.js:7873)
    at updateClassComponent (react-dom.development.js:7850)
    at beginWork (react-dom.development.js:8225)
    at performUnitOfWork (react-dom.development.js:10224)
    at workLoop (react-dom.development.js:10288)
    at HTMLUnknownElement.callCallback (react-dom.development.js:542)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:581)
    at invokeGuardedCallback (react-dom.development.js:438)

react-dom.development.js:9747 The above error occurred in the <RawDataModal> component:
    in RawDataModal (created by CubeView)
    in div (created by CubeView)
    in CubeView (created by SwivApplication)
    in main (created by SwivApplication)
    in SwivApplication

Favourite data cubes

I would like mark selected data cubes as my favourites. Favourites data cubes should be easily accessible from main page.
Nice to have: direct url to the page with my favourites.
Related issue: #71

Selecting row in table view is broken

Clicking on a table row to filter on a dimension's value fails with following error in browser console:

Uncaught TypeError: Cannot read property '__nest' of undefined
    at getFilterFromDatum (table.tsx:59)
    at Table.webpackJsonp../src/client/visualizations/table/table.tsx.Table.onClick (table.tsx:150)
    at Table.onClick (createPrototypeProxy.js:44)
    at Scroller.webpackJsonp../src/client/components/scroller/scroller.tsx.Scroller.onClick (scroller.tsx:270)
    at Scroller.onClick (createPrototypeProxy.js:44)
    at HTMLUnknownElement.callCallback (react-dom.development.js:542)
    at Object.invokeGuardedCallbackDev (react-dom.development.js:581)
    at Object.invokeGuardedCallback (react-dom.development.js:438)
    at Object.invokeGuardedCallbackAndCatchFirstError (react-dom.development.js:452)
    at executeDispatch (react-dom.development.js:836)

Ability to resize dimensions and measures panels

For data cubes with long list of dimensions and short list of measures (or long list of measures and short list of dimensions) the screen viewport is not fully utilized. The panel boundary between dimensions and measures panels should be draggable.

Use either Promise, Q.Promise or other Promise library consistently

During the upgrade to Plywood 0.17.x some Q.Promise uses have been refactored to native ES6 Promises because Plywood moved to native Promises.

Some of the uses have been left intact to not extend the dependencies upgrade to a big refactoring.
Because of this the usage of Promises is now inconsistent.

We should evaluate the options to use either native Promise (which are quite basic), Q.Promise or maybe even BlueBird - which are far more advanced and can accept or replace native Promises.

Export data cube definition from UI

New feature to export current data cube definition from UI as JSON (format compatible with mkurl payload). Right now it is really hard to figure out payload for mkurl endpoint. With this feature the existing visualisation would be used as a prototype for mkurl.

UI design proposal:
New action "Display view definition" under "Share link" menu, cube definition should be presented in the new modal window (the last element in the menu).
Change "View raw data" into "Display raw data".

Nice to have:
JSON syntax highlighting.

Description for dimensions and measures

In order to understand the meaning of dimensions and measures
As an analyst (or any business user)
I want to have rich description of the dimensions and measures presented directly on data cube page.

• Description should be defined in the configuration file (field name: description)
• Description should be defined as Markdown or plain text
• Description should be easily accessible by users on the left panels with strong visual indicator (e.g. question mark icon), activated on click
• Description is not searched

Line chart's Legend tab does not behave correctly when split on filtered dimension

Opening the following link with Turnilo started in wikipedia example mode presents an hourly line chart filtered for 7 "Country ISOs" and split by "Country ISO" with Legend tile on the right.

There are following issues:

• Legend contains only 5 elements instead of 7 with first 5 selected
• deselecting element in Legend removes corresponding value from filter but it should only hide the serie from line chart