malloydata / malloy-vscode-extension

The Malloy Visual Studio Code extension facilitates building Malloy data models, querying and transforming data, and creating simple visualizations and dashboards

Malloy

Malloy is an open source language for describing data relationships and transformations. It is both a semantic modeling language and a querying language that runs queries against a relational database. Malloy currently supports BigQuery, Postgres, and DuckDB.

We've built a Visual Studio Code extension to facilitate building Malloy data models, querying and transforming data, and creating simple visualizations and dashboards.

Click here to try Malloy in your browser!

---

Install the Visual Studio Code Extension locally

To write your own Malloy models, use the VS Code Extension, currently available for Mac, Linux, and Windows machines.

1. Download Visual Studio Code: Download Visual Studio Code

2. Add the Malloy (pre-release) extension from the Visual Studio Code Marketplace: Open VS Code and click the Extensions button on the far left (it looks like 4 blocks with one flying away). This will open the Extension Marketplace. Search for "Malloy" and, once found, click "Install"

3. Download and unzip the Sample Models (models + data).

4. Open the samples folder in VS Code. In VS Code, go to File > Open Folder... select malloy-samples/duckdb > Open. DuckDB is built into the extension so you're ready to run these.

5. Start with 1_airports.malloy in the FAA dataset. This is a sub-sample of the NTSB Flights dataset. In the editor pane, above source: airports, click the word "Preview" to run a SELECT *, and click the word "Run" above any query object to run it (see gif below for example).

To get to know the Malloy language, follow the Quickstart guide and/or continue through the numbered models in the FAA directory.

Note: The Malloy VSCode Extension tracks a small amount of anonymous usage data. You can opt out in the extension settings. Learn more.

Join the Community

• Join our Malloy Slack Community! Use this community to ask questions, meet other Malloy users, and share ideas with one another.
• Use GitHub issues in this Repo to provide feedback, suggest improvements, report bugs, and start new discussions.

Resources

Documentation:

• Malloy Language - A quick introduction to the language
• eCommerce Example Analysis - a walkthrough of the basics on an ecommerce dataset (BigQuery public dataset)
• Modeling Walkthrough - introduction to modeling via the Iowa liquor sales public data set (BigQuery public dataset)

YouTube - Watch demos / walkthroughs of Malloy

Contributing

If you would like to work on the Malloy VS Code extension, take a look at the instructions for developing.

To report security issues please see our security policy.

Malloy is not an officially supported Google product.

Syntax Example

Here is a simple example of a Malloy query:

query: table('malloy-data.faa.flights') -> {
  where: origin ? 'SFO'
  group_by: carrier
  aggregate:
    flight_count is count()
    average_flight_time is flight_time.avg()
}

In SQL this would be expressed:

SELECT
   carrier,
   COUNT(*) as flight_count,
   AVG(flight_time) as average_flight_time
FROM `malloy-data.faa.flights`
WHERE origin = 'SFO'
GROUP BY carrier
ORDER BY flight_count desc         -- malloy automatically orders by the first aggregate

Learn more about the syntax and language features of Malloy in the Quickstart.

malloy-vscode-extension's Issues

Revive the Malloy help panel

This is a brief summary from a separate discussion.

RP 753 removed the help panel from the extension because there were a few alternatives(e.g., hover messages) to show hints to customers at the time. A recent request is to bring back the Malloy icon and the help panel so that more content can be displayed in the help panel, and/or consolidating some other panels(connection panel). This issue is to track the work to revive the help panel and makes it generally helpful to customers.

Related PRs

PR 705

Decouple language server and worker from Malloy extension

The current implementation packages language server and worker directly, which is undesirable because it's not easy to do lazy loading in the vscode extension framework. Instead, we want to decouple the interplay through IPC messages:

• create an IPC communication mechanism(e.g., json over standard I/O) to allow extension to talk to lang server, etc.
• generalize it so that multiple other components can reuse (e.g., woker).

Keyboard shortcut to run queries

Should probably run currently highlighted query or query where cursor currently is

Styles from json import are not applied in Windows

Looks like the logic used to read the styles file does not correctly handle the format in the case of a windows path.

Web Extension limited in Safari, Firefox

Safari doesn't support web workers spawning web workers, so DuckDB cannot load its worker. We can still do syntax highlighting and outlines, but schemas, validation and running queries will not work.

Running queries is not working in Firefox, but enough other things work, like the schema view and validation, to make me think that this should fixable.

keytar is deprecated

Keytar as a separate package has been archived, VS Code is moving to use Electron's safeStorage, we should probably do something similar (one less binary dependency would be great, too)

Canceling a running query kills extension

Pressing cancel when running a query consistently kills extension for me, resulting in all future queries erroring like:

Reproduce using a longer-ish running query - press Run Query then press Cancel in progress bar in lower right. This query lets me do it consistently. Exception is:

Exception has occurred: Error: Request malloy/queryPanel failed with message: Webview is disposed
  at handleResponse (/Users/bporterfield/dev/malloydata/malloy-vscode-extension/dist/server_node.js:2100:40)
    at processMessageQueue (/Users/bporterfield/dev/malloydata/malloy-vscode-extension/dist/server_node.js:1938:13)
    at Immediate._onImmediate (/Users/bporterfield/dev/malloydata/malloy-vscode-extension/dist/server_node.js:1924:11)
    at process.processImmediate (node:internal/timers:466:21)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)

Appears to be related to recent change to combine worker and server, trying a build from before that merge did not reproduce the error.

Better dark mode support for notebook rendering

Currently we're not setting any styling, we can probably do something like we're doing with the query panel.

Run code lens only works on last unnamed query

Because of ambiguity between cells running queries by index doesn't work like it used to.

Run code lens appears on `nest`s inside a `from()`

I need to double-check that this isn't an issue with my branch, but I don't think so.

m4 wording changes in VSCode

Change 'Run Query' to 'Run' but only if @mtoy-googly-moogly agrees

• If checked @mtoy-googly-moogly agrees
  

In Schema view Change 'QUERIES' to 'VIEWS'

Protect against SQL injection in table names in postgres

The query to get the schema for a table in postgres doesn't properly quote the table names and is vulnerable. Both the schema and the table name are pushed into an SQL template without proper quoting.

Postgresql Foreign Data Wrappers (FDW) are not recognized when using the SQL syntax

Hi,
So, in my Postgres database (pis) I quote foreign tables from a MySQL database using the FDW_MySQL extension. Works great.
One of the tables is Tags in the schema zxt.

Malloy seamlessly recognizes the foreign tables when using its table syntax.
For example:

source: tagbas is pis.table('zxt.Tags') extend {...}

That is GREAT.

However, when I try inserting a raw SQL block such as:

source: tagpoibas is pis.sql("""
    SELECT
        name AS nam,
        id,
        pid
    FROM zxt.Tags
    WHERE pid in (SELECT id FROM Tags WHERE kuq_plc is TRUE)
""")

Malloy complains and throws an error:

Invalid SQL, Error fetching schema for md5:/...

I have verified the query: it works fine in Datagrip.

SQL code highlighting is broken

Once upon a time, I think, we highlighted SQL Inside of Malloy properly. Something like:

source: MOCK_DATES is duckdb.sql(
"""SELECT
  ts::DATE as MOCK_DATE_COLUMN
  FROM generate_series(
    TIMESTAMP '2000-01-01',
    TIMESTAMP '2030-01-01',
    INTERVAL '1' DAY) AS t(ts)
""")

There are two problems.

1. Someone (I suspect me) broke this.
2. There was no test to stop me from breaking this.

notebooks don't work until you save the file

Once I save this to a file, the error goes away.

handle "Query exceeded limit for bytes billed." for BigQuery

When folks hit this they don't know what to do next. It shouldn't be hard for us to catch this error and suggest they open connection settings and modify.

Confusing error reporting on file that does not exist

What happens?

In the Github browser-based VSCode environment, if any source contains an invalid filepath, then all sources will report errors. See screenshot below:

To Reproduce

Load this notebook in github.dev: https://github.dev/malloydata/learn/blob/f31186f8d7598eac8d63c9c009228ddb82445f3e/malloy_cardio.malloynb

Or, in the Github.dev environment create a malloy file with two sources, one that points to a valid file, and one that points to an invalid filename. Both sources will show errors.

OS:

macOS

Malloy Client:

web-based VSCode

Malloy Client Version:

v0.2.1694536284

Database Connection:

DuckDB

Need to decide how to handle database configuration and workspaces

See #364 (comment)

Currently we save to both the global and workspace config. There are differing opinions about whether connection config should be global (simpler to understand) vs workspace-specific (more fine-grained control?).

"Malloy Schema" is broken in the web extension

Load any model on github.dev. Doesn't show the schema for any of the malloy files.

Surface Query Run Time in VS Code UI

it will be nice to see how long a Query has took

Editing an imported file doesn't force a recomple when hittting 'Run Query'

When looking at the schema, running a query, etc. We should make sure we recompile first if one of the dependent files has changed. I end up making a small change to the file I'm working in to force a re-compile.

Assigning @mtoy-googly-moogly to make sure this info is available.

Code complete does not work if schema has spaces in column names

What happens?

In VS Code "code complete" (Intellisense) does not seem to work if the schema has spaces in the column names. The autocomplete adds the value, but not the surrounding backticks

To Reproduce

Create a new source in FAA flights.malloy:

source: flights2 is flights extend {
  dimension:
    `The Month` is dep_time.month
}

Then write: run: flights2 -> { select: T and select the autocomplete option:

This results in:

OS:

mac OS

Malloy Client:

Local VS Code

Malloy Client Version:

v0.3.1708023985

Database Connection:

DuckDB

Customer reported slowness

@anikaks can fill in the details.

I have the file: https://gist.github.com/mtoy-googly-moogly/593fbd9f291461fddb3634131f07863f which is a several thousand line pick statement, and the lag between a keystroke and the syntax highlighting updating on my mac is around five seconds.

Changing connection configuration for connection named "bigquery" is wiped out on VSCode restart

What happens?

when I click test everything works, and I even can pull some table column names with the source: test_source is bigquery.table('my_dataset_dw.staging.pendo_page_definitions') command as can be seen on the 2nd screenshot. However when I close my VS Code and open this project again, open Malloy connections, all the connection info I entered previously disappears, leaving me only with connection name bigquery

To Reproduce

see above

OS:

dont know

Malloy Client:

local VSCode, web-based VSCode

Malloy Client Version:

latest

Database Connection:

BigQuery

In Dark mode, SQL is pretty much unreadable.

Caleb, want to take a look?

Preview generates errors when SQL blocks are invloved

source: orig is table('bigquery:bigquery-public-data.google_cloud_release_notes.release_notes') {
  dimension: custom is 'CUSTOM'
}
query: first100 is orig -> { project: *; limit: 100 }
sql: to_sql is {
  connection: "bigquery"
  select: """
    SELECT *
    FROM (%{ ->first100 }% )
  """
}
source: back_to_malloy is from_sql(to_sql)
query: back_to_malloy -> { project: * }

Running the end query works, clicking "Preview" on "back_to_malloy" gets an error. SQL Blocks sit in the model in a weird way and however preview is generating a model to run the preview query, it is hitting into this problem.

Web extension cannot load worker under vscode.dev

vscode.dev incorrectly escapes the URL for the worker thread when the extension is loaded:

This is being tracked by microsoft/vscode#168674 on the VS Code side.

}%all alone context breaks parser

Currently this source file causes the malloy parser to throw an exception

%}

And "the output" is ...

[Error - 11:37:07 AM] Request textDocument/codeLens failed.
  Message: Request textDocument/codeLens failed with message: EmptyStackException
  Code: -32603 

Schema doesn't appear to be loading in the Web Extension

It appears that the web extension is not displaying Malloy Schema information.

maybe extension should show docStrings for objects when hovering?

Now that we have tags and there is a way to write docStrings, maybe the extension should show them on a hover?

Copy to Clipboard does not copy backticks if the field has a space in it

What happens?

In VS Code schema viewer, clicking the name of a field that contains a space copies the field name, but not the surrounding backticks

To Reproduce

Create a new source in FAA flights.malloy:

source: flights2 is flights extend {
  dimension:
    `The Month` is dep_time.month
}

Click on the Schema codelens for this source, then click on The Month dimension, and paste it into the text editor. It pastes as:

The Month

which does not compile. Would be great to have it paste as:

`The Month`

OS:

mac OS

Malloy Client:

Local VS Code

Malloy Client Version:

v0.3.1708023985

Database Connection:

DuckDB

Webview not re-opening

If I close the Malloy webview and try to run another query, I get error messages like this._token.cancel is not a function or Webview is disposed. The only way I know to fix the problem is reload the window.

If I'm understanding this problem correctly, it should be possible to solve it by checking whether the webview is still available and creating a new one if not. (I encountered similar behavior while developing the RelationalAI VS Code extension and solved it in that way.)

SQL syntax highlighting is applied in Turducken blocks with nested queries

When writing a Turducken block, which can now be closed by either }% or }, the syntax highlighter appears to greedily search for the single closing bracket character, }, without regard for opening parentheses. As a result, Malloy highlighting is only applied to certain parts of nested queries within Turducken blocks and SQL highlighting to other parts.

Beginning on line 20, SQL highlighting is applied until the end of the larger SQL block denoted by """.

Add support for remote URLS

See https://github.com/malloydata/whatsnext/blob/main/wns/WN-0010-module-support/wn-0010.md for reasoning why

Renderer breaks with multiple SQL statements in a single MalloySQL cell

What happens?

Random edge case bug -- if I have two MalloySQL statements in a notebook cell and execute the notebook, the results come back as JSON, not rendered:

To Reproduce

Create a notebook, add a MalloySQL cell with 2 SQL queries, and execute the notebook. For example, I used the flights dataset, and added a cell like so:

-- connection:duckdb
SELECT * FROM %{ flights -> top_destinations }%;
SELECT * FROM %{ flights -> by_month }%;

OS:

macOS

Malloy Client:

local VSCode

Malloy Client Version:

Pre-release

Database Connection:

DuckDB

Commenting code now use /* */

I much prefer commenting with // as it allows you to do partial uncomments.

Was this intentional?

We can no longer copy field names from the 'schema' view.

All the names are gray'd out and we can no longer click them.

Errors cascade

When running all, errors cascade and aren't necessarily colocated with the cell that caused them.

The execution should stop at the errored cell and not carry forward.

# comment after measure definition causes Run|Show SQL to appear in wrong spot

Update syntax code for expanded quoted identifiers

malloydata/malloy#944

Used to be only numbers and letters in backquoted identifiers .. now pretty much anything.

malloy icon missing in left bar when opening a file

% code ~/trusted/samples/duckdb/faa/flights.malloy

% code ~/trusted/samples/duckdb/faa

Then click on flights.malloy

Move notebook cell execution into worker.

Cell queries execute in the main extension thread, which means they can crash the extension.

Consolidate more worker functionalities into Malloy language server

#113 sets up a way to offload Malloy semantics heavy liftings such as model building to the Malloy (vscode) language server. This issue tracks the effort to consolidate more worker functionalities into Malloy language server:

• worker's createRunnable() creates a data model, which can be done in the language server. This may apply to other loadXYZ() calls.
• We could pass the language server object down to worker so that we may not need to create separate message handlers, as most of the work can be performed through talking to the server.

Files with `.malloysql` extension do not allow tagged sections

What happens?

I want to save the result of a Malloy view into a Postgres 13/Postgis database.
As per the documentation:

1. I created a .malloysql document
2. I began the document with >>>malloy to define a Malloy code block
3. I also defined below a >>>sql code block.

However, Malloy immediately complains about the first block opener:

mismatched input '>' expecting {<EOF>, AGGREGATE, CALCULATE, DIMENSION, GROUP_BY, JOIN_CROSS, JOIN_ONE, JOIN_MANY, MEASURE, NEST, PROJECT, QUERY, RUN, SELECT, SOURCE, VIEW, ALL, AVG, CAST, COUNT, DAY, EXCLUDE, FALSE, HOUR, IMPORT, IS, MAX, MIN, MINUTE, MONTH, NOT, NOW, NULL, PICK, QUARTER, SECOND, SOURCE_KW, SUM, TABLE, TRUE, WEEK, YEAR, HACKY_REGEX, SQ_STRING, DQ_STRING, BQ_STRING, DOC_ANNOTATION, ANNOTATION, '(', '{', '-', ';', LITERAL_TIMESTAMP, LITERAL_HOUR, LITERAL_DAY, LITERAL_QUARTER, LITERAL_MONTH, LITERAL_WEEK, LITERAL_YEAR, IDENTIFIER, NUMERIC_LITERAL, INTEGER_LITERAL, '"""'}
  | >>>malloy

What am I doing wrong?
S.

To Reproduce

1. In VS Code, create a tst.malloysql document
2. Open a Malloy code block by >>>malloy
3. Immediately, Malloy throws an error.

OS:

Ubuntu 20.04

Malloy Client:

self-hosted CodeServer bare metal on Ubuntu 20.04

Malloy Client Version:

VS Code Extension v0.3.1714196298 (pre-release)

Database Connection:

Postgres 13

When commenting out code in a MalloySQL code block use `--`

The dash dash comment works in both SQL and Malloy. We should use that as the comment when commenting/uncommenting code in a MalloySQL code block.