runreveal / pql Goto Github PK

• Select license
• Add file license headers
• Add VSCode copyright snippets

Not sure if expected or not. I want to do some post-processing of columns that are selected.

runreveal_logs
	| project toLower(eventName), notEmpty(eventName)
	| limit 5

Are things like this currently supported? I couldn't quite figure out if they were or not with another syntax. This way I'm currently doing it I get this error:

pql: parse pipeline query language: 2:26: expected '|', got '('
pql: one or more statements could not be compiled

Keeping an issue open here to decide on the language/repository name and rename things as appropriate.

Needed for #8

From @abraithwaite:

Support for map column type would be helpful I think. Not sure the best way to do this, but the syntax for a map key access is: mapcol['strkey'].

In other projects, I use Nix to manage the CI steps (example). This guarantees that running nix flake check locally reproduces the CI build exactly, but puts Nix in a more prominent place in the build process, which may not be desirable.

Alternatively, we could use Nix just as a package manager and use it to install Clickhouse (or possibly Go as well) in CI.

The let operator allows users to assign variables to be used in queries. Example follows.

let n = 10;  // number
let place = "Dallas";  // string
let cutoff = ago(62d); // datetime 
Events 
| where timestamp > cutoff 
    and city == place 
| take n

Kusto docs:

https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/let-statement

Not sure what the best way to approach this problem is, but support for an LSP would be huge for obvious reasons.

Theoretically, this should be more straight forward than SQL because we have better context at the point of typing.

That is, context flows through the pipe, versus SQL SELECT a, b, c FROM X where we don't know the possible values for a, b, or c until the table X is specified.

Reference

Since we're already extracting CTEs anyway, we can just have a no-op that captures aliases at particular points of a pipeline.

Reference

We should figure out how to support parameterized queries for databases.

They have various distinct ways of being implemented that we should be able to support

We may be able to use the passthrough mechanisms plus some side-channel loading of parameters somehow, as an idea.

Part of #5

• Logical operators
• Numerical operators
• String operators

Somewhat a combination of function and binary operators, since the syntax is x in (y, z, ...).

From @ejcx:

One thing that might be helpful to add [...] is a CONTRIBUTING.md and information on the general how to add new operators and translation target syntaxes.

Ensures correctness of generated queries beyond the manual review required in #3.

• true
• false
• null

Part of #5

Used to detect regressions in SQL output as changes are made. Should support a record mode.

How can we validate the generated query. For e.g.

Allowed Databases, Allowed Tables, Allowed Fields specific to each Table, on Where conditions checking for certain conditions, on limits checking the min and max limit value that the user can pass etc

If the user will be writing the query in PQL format. We want to validate for valid tables, but also limit the way the user queries instead of allowing them to query everything.

• Identifiers
• Quoted identifiers
• Pipes
• #15
• #18
• #19
• #20

• strcat should be converted to the SQL || operator
• isnull and isnotnull
• not
• others...?

• Tabular expression structure
• #23
• Parse count tabular operator
• Parse where tabular operator
• Parse take tabular operator
• Parse project tabular operator
• Parse summarize tabular operator
• Parse sort tabular operator

This seems very similar to the effort done by the PRQL team.

How does this project differ and have you considered joining forces instead of creating a new SQL competitor.

Syntax for ["foo"] or ['foo']. Details

Part of #5.

There are some extra string operators that have a variety of keywords and forms that can be investigated.

To preserve evaluation order, I'm inserting parentheses around all binary operators. However, for subexpressions that do not contain another binary operator in the resulting SQL, these parentheses are unnecessary.

Reference

Reference

Equivalent to sort by x | take y