kranf99 / kibella Goto Github PK

Currently, we can choose a bright/dark theme here:

The selected theme is saved inside a cookie. Instead of saving the selected theme inside a cookie, it must be saved inside the parameters of the dashboard. In this way, when the user request to load a specific dashboard, the theme switch automatically to the theme saved inside the loaded dashboard.

the v3 version brings many changes: The integrated help documentation needs to be re-done almost completely to match the new developments.

The label should be "Allow non-authenticated users to see this dashboard" instead of simply "shared".

Also: The checkbox shouldn't be inside the "share" menu option but rather inside the "save dashboard" menu.

when sharing a dashboard, it sometime interesting to prevent the users to see the raw data-table "below" a chart. Currently, each dashboard has a yes/no flag that says "Allow non-authenticated users to see this dashboard". The objective is to add a another yes/no flag that says: "Allow non-authenticated users to see the raw data-table below a chart".

The color of the legend in the dark theme in the "pie" chart and in the "Bar&Line" chart is not ok:

I should be black background and white text.

inside the file "interface/kibana.js", we can read:

  var secondSlash = window.location.pathname.indexOf('/', 1);
  var kbnPath = window.location.pathname.substr(0, secondSlash) || '/kibella';

This means that the value of kbnPath is "hard coded" to a directory named "kibella".
This is not good at all!

There is at least one place where the word "Elasticsearch" is mentioned in Kibella: in the page that lists the registered tables. See screenshot.

GOAL: Make filters on character fields work in SQL Server

CONTEXT: Any time we drill down on a character field in a visualization.

DETAILS:
Currently Kibella sends a filter on a character field by enclosing string values in DOUBLE quotes. However, double quotes are not considered valid by SQL Server to enclose strings...

This problem cannot be easily solved on the server side (except for very special --albeit common-- cases, namely SINGLE equality expressions of the form <field> = <value>) because the filter is applied via a general expression (called "query string" in Kibella). In fact, it is not easy to distinguish field names from field values in a general filtering expression, and decide whether double quotes should be replaced with single quotes or not.

PROPOSED SOLUTIONS:
I propose two possible solutions, the first one (done on the client side) would solve the problem for good, while the second one (done on the server side) would apply for very simple (but fairly common) cases.

The proposed solutions are:
S1) CLIENT SIDE: Have the Kibella client send string values enclosed in SINGLE quotes.
S2) SERVER SIDE: Replace double quotes with single quotes ONLY in field values for expressions of the simple form containing one single equality.
Note that expressions of the form <field1> = <value1> and <field1> = <value2> will NOT be tackled here.

EXAMPLES:
E1) INCOME table: when we filter on "class of worker" = "Private", we have the ambiguity to solve between a field name and a field value.
The part of the JSON string sent by the Kibella client to the server that is responsible of filtering looks something like this:

	"query":{
		"query_string":{
			"analyze_wildcard":true,
			"query":"\"class of worker\" = \"Private\""
		}
	}

If the proposed solution (S1) is implemented, the JSON string would look like this:

	"query":{
		"query_string":{
			"analyze_wildcard":true,
			"query":"\"class of worker\" = 'Private'"
		}
	}

and this would make the filtering on the value 'Private' automatically work in SQL Server.

from here:
https://github.com/dlumbrer/kbn_dotplot

From here:
https://github.com/nreese/kibana-time-plugin

This visualization must be re-made using plotly.
It should keep the same options as the original (e.g. change colors, display both bar and lines).
The original visualization did not support "drill-down": The "drill-down" functionality must be added.

Once this new visualization is done&tested, we'll get rid of the C3JS library to make a much lighter "kibella.js" file: i.e. C3JS (and all the linked libraries) must be removed from the dependency list.

Currently, Kibella works perfectly on the "standard" port numbers: i.e. port 80 for http and port 443 for https. ...But, if you are using any other port than the "standard" ones it fails.

More details:
You can see here that the connection to "http://localhost/kibella/db/?_=1578..." is not working:

...and this was to be expected because, in this example, kibella was installed on "http://localhost:4443".

Thus, you can manually edit the url (to add the port number) and you can see that it's working (i.e. "status code" is OK):

To summarize: Kibella "loses" the port-number for some requests to the back-end,
Would it be possible to fix that?

Sometimes we want to filter a visualization/dashboard on "all the values except that particular one".

Current situation:

1. We first need to click on the value we want to exclude -> This actually launches the request where that value is the only one included.
2. Then we need to apply the negation of the filter just applied in order to exclude the value.

Being able to go directly to step (2) would reduce user's waiting time.

Note that currently something similar is already implemented when filtering on a point coming from a multilevel aggregation (e.g. terms aggregation on education + range aggregation on age) since Kibella first asks us whether we want to apply all filters (i.e. on both the education dimension and the age dimension) and we are given the option to select only one filter.

Today I was playing around with Kibella's live version (http://vps714395.ovh.net/kibella) and found the bug that can be seen in the screenshot below, where we see the message "No results displayed because all values equal 0" on the visualization and the top-right corner.

However, the query (running a negative filter on 3 fields) does return records (as seen by: (i) the other visualizations show a graph; (ii) when running the SQL equivalent query directly on the SQLite database we get more than 80k records).

The SQL-equivalent query filter is:

race != "Black"
and sex != "Male"
and education != "Bachelors degree(BA AB BS)"

Note that I don't observe this problem in my local version of Kibella where I am using the interface version dated 02-Sep-2019 still available at http://download.timi.eu/Kibella/Kibella_UwAmp.zip.

On this version, I see the following results table:

see example screenshot:

In the Discover Tab, remove the internal variables (_index, _type) from the variables list?

from here:
https://github.com/datasweet-fr/kibana-datasweet-formula
...and remove all the "datasweet" publicity

Goal: Keep the user "interested" and already looking at some results when a big dashboard takes "too long" to load.

Currently the visualizations in the dashboard are shown all at once, only after all the visualizations have been run.
The proposal here is to show each visualization when the associated response file has been created.

This enhancement will imply a few changes:

• the interface should be prepared to receive the responses at different moments and process them
• the backend should send the responses as they become ready

Probably an important effort is needed on the interface side in order to process a partial response, since the response the interface currently processes contains the responses for ALL the visualizations present in the dashboard.

Once in a while I get a "No Living Connections" error as shown in the screenshot.

I observed the following:

1. It seems to happen erratically, e.g. for no specific query and after no specifically amount of time had passed. For instance, I observed it happening after just 20 seconds of a dashboard query running.

2. If we refresh the page after this error happens, the dashboard results are shown instantaneously (if cache is enabled for the queried table).

Note that the max execution time is set at 300 seconds (5 minutes) by the backend, using the following line in config.php:
ini_set('time_limit', 300);

Interface version: 02-Sep-2019 (Ref: date in http://download.timi.eu/Kibella/ page)

• fix the "Aggregation Display Options" -> "map type" option: only the "scaled circle markers" is working (but it does not scale anything).
• “change precision on map zoom” option is not to working (it's stuck on precision=2)

from here:
https://github.com/fbaligand/kibana-enhanced-table
or here:
https://github.com/seadiaz/computed-columns

The HTML widget allows to add any user-defined HTML code inside a dashboard.
The objective is to provide to the user-defined HTML code several informations:

• the current user login
• the current "drill-down filter".
• any other idea?
  This will allow to create a "HTML hooks" to other platforms (that are using REST API's).

Currently, to create a new visualization, you use a wizard in 4 steps:
Step1: select the vizualisation type:

Step2: click on the "new" button. Actually, the "open" button is useless: it's something that was initially there because of kibana but it cannot be used anymore now:

Step3: Select the table on which the aggregates/visualizations are computed:

Step4: design your visualization.

The objective of this "issue" is to completely remove the Step2: i.e. do not display any "new" or "open" button anymore: i.e. Always do "as if" the user clicked on "new".

1. Home buttons
   Home button should always be at the top of the page via the Kibella icon.
   This means that the back button should be changed in the Administration page, which currently appears at the bottom left with the name "Home" (sometimes it's not even seen).

The state of the checkbox is not currently representing the real value stored inside the database.

The value of the check box is not saved inside the kibella.sqlite file (i.e. inside the column "_shared" of the "Objects" table).

more precisely the status is saved inside the "_shared" column inside the "objects" table

all dates coming from the back-end are UTC dates (and not "local time zone" dates).
So they must be always printed on screen using the "date.toUTCString()" method.

based on this plotly char:
https://plot.ly/r/waterfall-charts/

on the old bar chart, the popups are:

on the new bar&line chart, the current popups are ugly:

The objective is to get the new popups (on plotly) that looks like the old (more beautiful) ones. Also, make sure that the popups also works with the dark theme because, currently, we have:

Furthermore, the X axis does not display all expected value (see screenshot below: the "10" is missing).

cannot visualize the dashboard in an iframe

Observed problem: Pasting on a web browser the URL associated to the "Share a Read-Only link" option generated by clicking on the "Share Dashboard" icon at the top-right of a dashboard sends us to the login page... (instead of showing the dashboard, without logging in)

when selecting a rectangle on a heatmap (openStreetMap) the selection is not perfect: some points are still displayed outside the selected area.

Currently when we open the Settings tab, we see the word "Index" or "Index Pattern" to refer to what we call "Table", and this may cause some confusion to the user.

So I think it is better name what we select as input data to work on as "Table" instead of "Index"/"Index Pattern".

Currently, when we want to share a dashboard, we receive a very long URL.
...and long URLs are boring/annoying. I want short URL's!

So, the idea is to create 2 PHP functions to "shorten" the URL's.
These 2 functions are then called from the angular.js interface accordingly.

• function 1:
  a PHP code that receives the long URL a send back a short URL that "points" to function 2.
• function 2:
  a PHP code that receive a short URL, expand the URL and re-directs towards the expanded (long) URL.

These 2 functions are using internally a small SQLite database with one "conversion" table. The 3 columns of this "conversion" table are:

• "MD5": primary key: md5 of the long url
• "Long_URL"
• "Last_Usage" (date of last conversion, or date of creation of this row)

More details:

• the "function 1" creates an URL that is composed of 2 parts:
  (1) points to a php code that is running function 2
  (2) has one parameter that is the md5 of the long url
• the "function 2 does the following:
  (1) use the received parameter to run:
  select LONG_URL, LAST_USAGE from CONVERSION where MD5=<received parameter>
  (2) update the "Last_Usage" column each time a convertion is performed.
  (3) redirects to the LONG_URL found in the CONVERSION table.

In previous versions of Kibella, a calendar used to appear at the top of Visualizations and Dashboards in order to easily filter on date fields when creating visualizations.

In fact, since dates must be stored in seconds in order for them to work correctly in the application, it is not easy to filter on dates using the free text box, and that's when the calendar comes handy.

I am basing my assumption that the calendar existed on the following:

• On 18-Jul-2016 I wrote among my task notes that a calendar appeared on dashboards even when no date field was present in the analyzed table.
• I created a couple of queries that I use in unit tests that have a filter on a date field.

Visualization of Other and Missing terms buckets

It's not possible to define "other" and "missing" buckets for terms aggregation. The "other" bucket shows the number of documents that didn't match the top N values retrieved by the terms aggregation. The most obvious use case for this is the pie chart, where you expect the slices to represent 100% of your documents. Turning on the "other" bucket allows you to achieve just that. On the other hand, the "missing" bucket counts the documents that do not have a value for the field in question.

recode pie chart with plotly

It would be nice to add a tooltip text to indicate what the different buttons do (e.g. the buttons on the left ribbon and, in the dashboard view the two buttons at the top right of each visualization).

More precisely:

On the left ribbon, the tooltips contains, in order:
"Home" (for the Kibella icon), "Discover", "Visualize", "Dashboard", "Settings"

For each bar and each line, the user can select which Y axis (left or right) it is using.
More precisely:

• by default, all bars&lines are using the left Y axis.
• for each bar or line, i'd like a new check box that say "use right Y axis" (instead of the default, which is the left axis).

The final objective is to be able to create a chart such as this one:

In the above chart:

• bars in green:
  different counts: "select count(*) from income group by education."
  these bars are linked to the LEFT y axis (that goes from 0 to 90000).
• line in red:
  different averages: "select mean(table_income_amount)*100 from income group by education."
  this line is linked to the RIGHT y axis (that goes from 0 to 35).

When navigating through Kibella I encountered the following error message once in a while (I am not sure what generates it).
It happened already in all the 4 tabs (Discover, Visualize, Dashboard, Settings).

A refresh of the page (F5) makes the error window go away.

System: observed at least in Firefox 67.0.4 (64-bit) under Windows 10 (64-bit), using uWAMP 3.1.0 (downloaded from timi.eu in Jun-2019).

in Sankey and other plots it is sometimes impossbile to order items. For example, year is ordered "2015-2016-2018-2017-2019", which makes data difficult to interpret.

the login page is nearly as big as the main "index.js"!
use webpack on the login page to reduce weight.

Context:

1. We have been working on different objects (Searches, Visualizations or Dashboards) on a particular a table (e.g. Table A).
2. We change the default table to work on in the Settings tab (e.g. Table B).
3. We go back to the Discover, Visualization or Dashboard tab.

Currently, after step (3) we see the latest object that was open in any of those tabs whose results correspond to the previous table A I have been working on.
Instead, I would like to see the landing page to create a new object since I am now working on a new table B.

The checkbox named "Enable the cache for the results run on this table?" is not working properly: i.e. in the POST request sent to the backend, the value of the parameter "enableCache" is always missing.

upgrade the lodash, angular, jquery, etc. libraries and check that nothing is broken.

Use a better keyword for the plots where the X-axis is defined by a categorical variable.
"Terms" comes from the Elasticsearch world, but it does not really convey that we are doing a plot on a categorical variable.

The keyword "Groups" should be preferred.

After I click the "save" button inside the HTML visualization, the HTML code always reverts back to "<i>italic</i> and <b>bold</b>".

Just before I click "save":

After, I click the "save" button, the HTML reverts back to "<i>italic</i> and <b>bold</b>".

I just looked at the post-data sent to the back-end: the data does not contain the HTML text to save (i.e. the "visState/params" is empty): i.e. There is an error in the js code.