mwater / mwater-expressions Goto Github PK

Call it to text. Converts enum to a string. Name is "Convert to text"?

Seems like a good usecase: immutable and passed to all children.

Functions that are aggregate but also have a condition

This implies that we can know the ancestry of a row. Will need new function to row.

It's tricky to evaluate within for an arbitrary expression. But this is needed with "within WaterAid" function of https://github.com/mWater/mwater-app/issues/632

When listing the population data for an admin boundary in a pivot table it defaults to the Total operator. However this sums up the population allocated for the level as well as all the levels below it, leading to a much inflated figure. Selecting 'Max' is a workaround for at least the second-lowest and lowest levels, but it's not intuitive for users to select that.

We need a more intuitive alternative for bringing through only the level-appropriate pop data.

Allow joining to another table, using location, calculating an aggregate value with a filter.

4.9s/100k vs 3.5s without cosine adjustment

select name, code,
(select count(*) from entities.household as hh 
  where 
 hh.location && st_expand(wp.location, 1000/cos(ST_YMin(ST_Transform(wp.location, 4326)) / 57))
 and
 st_dwithin(
	 wp.location,
	 hh.location, 
	 1000/cos(ST_YMin(ST_Transform(wp.location, 4326)) / 57)))

from (select * from entities.water_point as wp limit 100000) as wp

Faster (3.6s):

select name, code,
(select count(*) from entities.household as hh 
 where 
 hh.location && st_expand(wp.location, 1000/cos(ST_YMin(ST_Transform(wp.location, 4326)) / 57))
 and
 st_distance(
	 wp.location,
	 hh.location) <
	 1000/cos(ST_YMin(ST_Transform(wp.location, 4326)) / 57)
)
from (select * from entities.water_point as wp limit 100000) as wp

ExprValidator.js line 76

Should only contain:

toTable, type? (1-1, 1-n, n-n, n-1?) and either:

fromColumn, toColumn

or

jsonql: which is expr with {from} and {to} aliases.

When a user wants to see a date between two dates and one is a date data type and the other datetime, we should allow this to work. In these instances can we convert the datetime to a date, or else use the midnight time or similar?

How can duality of

a) raw geometry columns in database and needing those for mapping
b) geojson for client use

be maintained? what should the datasource return?

For single joins. Used for entity selection.

Three types: literal, non-aggregate (better name) and aggregate.

Only aggregate and non-aggregate can't be combined. Any other combination goes to the bigger one (e.g. aggregate).

Cleaning should do what? e.g. clean aggr to non-aggr? clean non-aggr to aggr?

aggr expressions (sum, etc) take literal or non-aggr. They produce aggrs.

Problem: aggr expressions can contain group by clauses! You just need to group on them... But it has the same effect as min or max of it, so ignoring.

Since each compiled expression is different, it never caches.

See https://github.com/mWater/mwater-server/issues/203

Several users requested. Takes an argument, optionally a second one to use where clause?

thisyear, lastyear, thismonth, lastmonth, today, yesterday, last7days, last30days, last365days

These are all new ops that only work on date and datetime fields. They should compile to jsonql that is a simple literal >= "2014-05-31" for example, depending on the current date when compiled.

Orderings use raw database column which adds complexity and prevents abstracting the database.

• Fix and doc in mwater-expressions
• Remove getOrdering from Row instance
• Change in mwater-vis compiler
• Correct in database for adhoc table
• Review mwater-forms
• Test adhoc visualization
• Release server and portal and blue

Should have expr property instead. Having as type prevents using enums as values since we can't specify enumValues.

• Change in definition
• Leave in expr handling but deprecated
• Change schema builders
• Property editor should have as extra control, not option
• Change exprType to expr in features of PropertyListComponent
• Change all existing indicators with expr properties
• Change aggregations in indicators to require type and enumValues?
• Prevent selecting in indicator calc builder
• Restrict calculations in forms to numbers

enum[] types are relatively common (multi-check questions). They can be a section of booleans instead but it is slightly awkward to use as you can't do one condition with multiple values.

ops needed:
= any to find if any intersection (&& if arrays, ?| with text[] rhs for jsonb)
= all to find if LHS contains all of RHS (@> if arrays, ?& with text[] rhs for jsonb))

RHS would need to be a literal (or use json_array_elements_text?).

{ type: "score", input: <expr of enum/enumset>, scores: { } }

0 if null

Use CSS instead

When adding a "Number of values in" filter for a multichoice question with a less than operator does not count entirely blank responses.

https://portal.mwater.co/#/dashboards/80a95dcf7b264c3b9d59ef51872ee8dd

• Add Filter and takes boolean

Add table should return new copy

Allow table and fields to have "deprecated" true. Should not be displayed in any selection ui but still work if present.

• Add to schema wiki definition
• Add to properties definition
• Modify scalar expr picker to support
• Add to table picker
• Add to schema builder for properties
• Add to schema builder for joins to deprecated tables

Make an object called PriorityDataQueue which takes a DataSource as a constructor parameter and a concurrency. It has one function: createPriorityDataSource which takes a parameter priority (integer). That returns a DataSource (a subclass, probably called PriorityDataSource) that looks like a data source but passes all calls to the original data source in priority order (highest first).

Recursive ones should null

Keeping separate is painful code-wise

When cases in expressions are of different data type, currently we are only looking at the first one. This causes issues when we have different types.
eg. text type in case 0 and enumset in case 1.

If the second condition is met, the enum values are not transformed

Right now is a special expr type and also a getExprType type of count.

• Make async
• Support aggregation
• Implement all functions

Aggregation takes N rows and produces one value (row?). i.e. does aggregation take N rows and produce 1 value?

To reproduce, create an expression with an if/then. Put in a value for the 'then' box, then immediately click on the '+if' to create another if case for the same statement. The value you entered disappears.

I noticed that if you enter the value, click on some dead space in the box (making the cursor disappear from the box), then +if, the value stays.

Switch to or using @>

It tries to look up value from the row, rather than evaluating the expression

date - date
datetime - datetime

• add expressions
• compile
• interpret
• ui