paulourio / bqlang

ZetaSQL tooling, mainly for BigQuery.

bqlang - tooling for ZetaSQL

I develop this tooling mainly for use with BigQuery.

Formatter - bqfmt

Status support

Supported features

• Qualify

• IS DISTINCT FROM

• Consecutive ON ...

• JSON type

  The JSON type is partially supported is select column lists, but not in other places.

• Allow dashes in table name (ie, FROM project-name.dataset.table without escaping)

• Create view with column list (CREATE VIEW vw(field1, field2))

• Remote Functions

Types

• Simple types: BIGNUMERIC, BOOL, BYTES, BYTES, DATE, DATETIME, FLOAT64, INT64, INTERVAL, NUMERIC, STRING, TIME, TIMESTAMP

• ARRAY

• STRUCT

• GEOGRAPHY

• Templated types

  Not supported ideally, but it works on my machine. Currently, go-zetasql has a bug to parse these types but we have a workaround to make it work.

Literals

All literals for all types are supported.

Comments

Google's ZetaSQL parser ignores comments. Current experimental implementation of bqfmt tries the best to format maintaining comments at the closest position possible from the input. Formatting without comment is always idempotent, but formatting code preserving comments is not guaranteed to be idempotent.

Expressions

• Field access operator (expression.fieldname[...])
• Array subscript operator (array_expression[array_subscript_specifier])
• JSON subscript operator (json_expression[array_element_id], json_expression[field_name])
• Arithmetic operators (X + Y, X - Y, X * Y, X / Y, +X, -X).
• Bitwise operators (~X, X | Y, X ^ Y, X & Y, X << Y, X >> Y)
• Comparison operators (=, !=, <>, >, <, >=, <=, [NOT] LIKE, IS [NOT], IN, IS [NOT] DISTINCT FROM).
• Conditional expressions (CASE, COALESCE, IF, IFNULL, NULLIF, )
• Logical operators (AND, OR, NOT)
• EXISTS operator (EXIST(subquery))
• IN operator (search_value [NOT] IN value_set)
• IS operator
• Concatenation operator X || Y
• Function calls (SQL functions, UDFs, named arguments)
• Aggregate function calls (function_name([DISTINCT] args [...modifiers]) OVER over_clause)
• Window function calls (function_name([argument_list]) OVER over_clause)
• Subqueries

Statements

Data Definition Language (DDL)

• Statements
  • CREATE SCHEMA
  • CREATE TABLE
  • CREATE TABLE LIKE
  • CREATE TABLE COPY
  • CREATE SNAPSHOT TABLE
  • CREATE TABLE CLONE
  • CREATE VIEW
  • CREATE VIEW defined with column list with options (CREATE VIEW t(field OPTIONS(...))) - not supported by go-zetasql
  • CREATE MATERIALIZED VIEW
  • CREATE EXTERNAL TABLE
  • CREATE EXTERNAL TABLE WITH CONNECTION
  • CREATE FUNCTION
  • CREATE TABLE FUNCTION
  • CREATE PROCEDURE
  • CREATE ROW ACCESS POLICY
  • CREATE CAPACITY - not supported by go-zetasql
  • CREATE RESERVATION - not supported by go-zetasql
  • CREATE ASSIGNMENT - not supported by go-zetasql
  • CREATE SEARCH INDEX - not supported by go-zetasql
  • ALTER SCHEMA
  • ALTER TABLE
  • ALTER COLUMN
  • ALTER VIEW
  • ALTER MATERIALIZED VIEW
  • ALTER ORGANIZATION - not supported by go-zetaslq
  • ALTER PROJECT - not supported by go-zetaslq
  • ALTER BI_CAPACITY - not supported by go-zetaslq
  • ALTER CAPACITY - not supported by go-zetaslq
  • DROP SCHEMA
  • DROP TABLE
  • DROP SNAPSHOT TABLE
  • DROP EXTERNAL TABLE
  • DROP VIEW
  • DROP MATERIALIZED VIEW
  • DROP FUNCTION
  • DROP TABLE FUNCTION
  • DROP PROCEDURE
  • DROP ROW ACCESS POLICY
  • DROP CAPACITY - not supported by go-zetasql
  • DROP RESERVATION - not supported by go-zetasql
  • DROP ASSIGNMENT - not supported by go-zetasql
  • DROP SEARCH INDEX

Data Manipulation Language (DML)

• INSERT
• DELETE
• TRUNCATE TABLE
• UPDATE
• MERGE

Data Control Language (DCL)

• GRANT
• REVOKE

Procedural language

• DECLARE
• SET
• EXECUTE IMMEDIATE
• BEGIN...END
• BEGIN...EXCEPTION...END
• CASE [search_expression]
• IF
• Labels
• Loops
  • LOOP
  • REPEAT
  • WHILE
  • BREAK
  • LEAVE
  • CONTINUE
  • ITERATE
  • FOR...IN
• Transactions
  • BEGIN TRANSACTION
  • COMMIT TRANSACTION
  • ROLLBACK TRANSACTION
• RAISE
• RETURN
• CALL

Export and load statements

• EXPORT DATA
• LOAD DATA

Debugging statements

• ASSERT

BigQuery ML SQL

• CREATE MODEL

Extensions

Jinja2

• Template variable ({{ variable }})
• Template blocks
  • For loop ({% for expr in iterable %}...{% endfor %})
  • If-endif statement ({% if cond %}...{% endif %})
  • If-else-endif statement ({% if cond %}...{% else %}...{% endif %})
  • If-elif-endif statement ({% if cond %}...{% else %}...{% endif %})
  • If-elif-else-endif statement ({% if cond %}...{% elif ... %}...{% else %}...{% endif %})

Currently, templates should be replaceable by an identifier or query statement so that the resulting query is a valid ZetaSQL script. If you follow this rule, you can use quite a lot of templates without losing the ability to format the SQL code before rendering.