cybercentrecanada / jupyterlab-sql-editor

A JupyterLab extension providing, SQL formatter, auto-completion, syntax highlighting, Spark SQL and Trino

jupyterlab-sql-editor

Installation

Follow the installation instructions in CONTRIBUTING

Security Vulnerability Reporting

If you believe you have identified a security vulnerability in this project, please send an email to the project team at [email protected], detailing the suspected issue and any methods you've found to reproduce it.

Please do NOT open an issue in the GitHub repository, as we'd prefer to keep vulnerability reports private until we've had an opportunity to review and address them.

Shout-outs

Many thanks to the contributors of these projects:

• krassowski/jupyterlab-lsp
• joe-re/sql-language-server
• zeroturnaround/sql-formatter
• cryeo/sparksql-magic
• trino-python-client
• bloomberg/ipydatagrid
• widgetti/ipyaggrid

jupyterlab-sql-editor's Issues

Trino cell magic(%%trino) does not work while same query works with trino line magic(%trino)

When using %%trino it always errors out with syntax error
TrinoUserError: TrinoUserError(type=USER_ERROR, name=SYNTAX_ERROR, message="line 1:22: mismatched input '<EOF>'. Expecting: 'ALTER', 'ANALYZE', 'CALL', 'COMMENT', 'COMMIT', 'CREATE', 'DEALLOCATE', 'DELETE', 'DENY', 'DESC', 'DESCRIBE', 'DROP', 'EXECUTE', 'EXPLAIN', 'GRANT', 'INSERT', 'MERGE', 'PREPARE', 'REFRESH', 'RESET', 'REVOKE', 'ROLLBACK', 'SET', 'SHOW', 'START', 'TRUNCATE', 'UPDATE', 'USE', <query>", query_id=20220910_013206_00228_xf4wd)

using the same query with line magic works well which means the connections and other params are fine.

Inspecting the query being sent by %%trino , I see that the sql query is not even being passed to trino.

Logs

---------------------------------------------------------------------------
TrinoUserError                            Traceback (most recent call last)
/tmp/ipykernel_48/3498095607.py in <module>
----> 1 get_ipython().run_cell_magic('trino', '--output html', "select 'hello line magix' as test\n")

~/.local/lib/python3.7/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2470             with self.builtin_trap:
   2471                 args = (magic_arg_s, cell)
-> 2472                 result = fn(*args, **kwargs)
   2473             return result
   2474 

~/.local/lib/python3.7/site-packages/decorator.py in fun(*args, **kw)
    230             if not kwsyntax:
    231                 args, kw = fix(args, kw, sig)
--> 232             return caller(func, *(extras + args), **kw)
    233     fun.__name__ = func.__name__
    234     fun.__doc__ = func.__doc__

~/.local/lib/python3.7/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/.local/lib/python3.7/site-packages/jupyterlab_sql_editor/ipython_magic/trino/trino.py in trino(self, cell, line, local_ns)
    114 
    115         self.cur.execute(sql)
--> 116         results = self.cur.fetchmany(limit+1)
    117 
    118         columns = list(map(lambda d: d[0], self.cur.description))

~/.local/lib/python3.7/site-packages/trino/dbapi.py in fetchmany(self, size)
    556         result = []
    557         for _ in range(size):
--> 558             row = self.fetchone()
    559             if row is None:
    560                 break

~/.local/lib/python3.7/site-packages/trino/dbapi.py in fetchone(self)
    524         try:
    525             assert self._iterator is not None
--> 526             return next(self._iterator)
    527         except StopIteration:
    528             return None

~/.local/lib/python3.7/site-packages/trino/client.py in __iter__(self)
    612         # Subsequent fetches from GET requests until next_uri is empty.
    613         while not self._query.finished:
--> 614             rows = self._query.fetch()
    615             for row in rows:
    616                 self._rownumber += 1

~/.local/lib/python3.7/site-packages/trino/client.py in fetch(self)
    789         """Continue fetching data for the current query_id"""
    790         response = self._request.get(self._request.next_uri)
--> 791         status = self._request.process(response)
    792         self._update_state(status)
    793         logger.debug(status)

~/.local/lib/python3.7/site-packages/trino/client.py in process(self, http_response)
    534         logger.debug("HTTP %s: %s", http_response.status_code, response)
    535         if "error" in response:
--> 536             raise self._process_error(response["error"], response.get("id"))
    537 
    538         if constants.HEADER_CLEAR_SESSION in http_response.headers:

TrinoUserError: TrinoUserError(type=USER_ERROR, name=SYNTAX_ERROR, message="line 1:23: mismatched input '<EOF>'. Expecting: 'ALTER', 'ANALYZE', 'CALL', 'COMMENT', 'COMMIT', 'CREATE', 'DEALLOCATE', 'DELETE', 'DENY', 'DESC', 'DESCRIBE', 'DROP', 'EXECUTE', 'EXPLAIN', 'GRANT', 'INSERT', 'MERGE', 'PREPARE', 'REFRESH', 'RESET', 'REVOKE', 'ROLLBACK', 'SET', 'SHOW', 'START', 'TRUNCATE', 'UPDATE', 'USE', <query>", query_id=20220910_013656_00235_xf4wd)

Support for OAuth2Authentication for Trino magic doesn't work

Hello folks!

I'm testing the extension a little bit. For now I'm just trying to connect to a Trino database through OAuth2Authentication without testing the other features (autocompletion/syntax highlighting).
When I try to configure that auth I get a pickle error:

%config Trino.auth=trino.auth.OAuth2Authentication() -> ERROR:root:cannot pickle '_thread.lock' object

Any ideas on how I could deal with it?

Trino query 'DELETE' filed

I'm trying to execute 'DELETE' query by Trino plugin and It fails. I suppose, jupiterhub plugin add some 'limit' to my query and this behavior affect execution. 'SELECT' query works fine. Also the same issue I have gotten when have executed 'UPDATE' and 'INSERT' queries.

My query:
%%trino
DELETE FROM delta.my_schema.jupiter_delta WHERE lang = 'Python'

Output message:

TrinoUserError Traceback (most recent call last)
Cell In[51], line 1
----> 1 get_ipython().run_cell_magic('trino', '', "DELETE FROM delta.my_schema.jupiter_delta WHERE lang = 'Scala'\n")

File /opt/conda/lib/python3.10/site-packages/IPython/core/interactiveshell.py:2422, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
2420 with self.builtin_trap:
2421 args = (magic_arg_s, cell)
-> 2422 result = fn(*args, **kwargs)
2423 return result

File /opt/conda/lib/python3.10/site-packages/jupyterlab_sql_editor/ipython_magic/trino/trino.py:119, in Trino.trino(self, line, cell, local_ns)
116 elif not args.raw is True:
117 sql = f'{sql} limit {limit+1}'
--> 119 self.cur.execute(sql)
120 results = self.cur.fetchmany(limit+1)
122 columns = list(map(lambda d: d[0], self.cur.description))

File /opt/conda/lib/python3.10/site-packages/trino/dbapi.py:439, in Cursor.execute(self, operation, params)
436 else:
437 self._query = trino.client.TrinoQuery(self._request, sql=operation,
438 experimental_python_types=self._experimental_pyton_types)
--> 439 result = self._query.execute()
440 self._iterator = iter(result)
441 return result

File /opt/conda/lib/python3.10/site-packages/trino/client.py:765, in TrinoQuery.execute(self, additional_http_headers)
763 # Execute should block until at least one row is received or query is finished or cancelled
764 while not self.finished and not self.cancelled and len(self._result.rows) == 0:
--> 765 self._result.rows += self.fetch()
766 return self._result

File /opt/conda/lib/python3.10/site-packages/trino/client.py:780, in TrinoQuery.fetch(self)
778 """Continue fetching data for the current query_id"""
779 response = self._request.get(self._request.next_uri)
--> 780 status = self._request.process(response)
781 self._update_state(status)
782 logger.debug(status)

File /opt/conda/lib/python3.10/site-packages/trino/client.py:581, in TrinoRequest.process(self, http_response)
579 logger.debug("HTTP %s: %s", http_response.status_code, response)
580 if "error" in response:
--> 581 raise self._process_error(response["error"], response.get("id"))
583 if constants.HEADER_CLEAR_SESSION in http_response.headers:
584 for prop in get_header_values(
585 http_response.headers, constants.HEADER_CLEAR_SESSION
586 ):

TrinoUserError: TrinoUserError(type=USER_ERROR, name=SYNTAX_ERROR, message="line 2:2: mismatched input 'limit'. Expecting: '%', '*', '+', '-', '.', '/', 'AND', 'AT', 'OR', '[', '||', ", query_id=20230111_132511_00024_k3w8w)

THANK YOU!

Hi,

I want to thank you for creating this wonderful project. We recently created and open sourced https://github.com/fugue-project/fugue-jupyter which is heavily inspired by your work. Although we are not able to directly leverage this project because Fugue SQL has more syntax than standard SQL, we learned a lot from this project.

For now, we only enabled highlights, in the future we will also enable auto formatting and probably auto completion. You have all of them, that is amazing!

Han

Setting catalog and schema for a trino session

The Trino connection used by %%trino should retain catalog choice. Presently it is ignored and requires the user to fully qualify tables and schema in a query.

trino magic: verify connection argument for adding certificate file path

jupyterlab-sql-editor/jupyterlab_sql_editor/ipython_magic/trino/trino.py

Lines 65 to 72 in a22ca90

	self.conn = trino.dbapi.connect(
	host=self.host,
	port=self.port,
	auth=self.auth,
	user=self.user,
	catalog=catalog,
	schema=schema,
	http_scheme=self.httpScheme)

Can we add verify connection argument for adding certificate file path?

Allow user to specify Trino Connection object conn and cur

A user may want to specify their own Trino connection in a notebook and add this to the Trino configuration. Presently the Trino configuration does not allow specifying the conn and cur attributes.

Autocomplate problem in Win 10.

My problem is the Autocomplete is not working.
Thank.

How to install (ELI5) ?

Hi,

Thanks for this promising extension. However, i am a newbie and not able to get this installed.
I did a pip install jupyterlab-lsp jupyterlab-sql-editor and npm install -g sql-language-server
and I do see these installed doing a pip list

However, when i do %%sparksql , it cant find it

Can anyone please advice?

Thanks in advance!

What needs to be loaded or imported to get this to work?

Hi, I just installed this package, and I'm just interested in using the syntax highlighting within strings feature. It isn't clear to me what I need to do to load this package and get this feature to work in my notebook. Any help would be appreciated. Thank you.

Feature Request: Spark SQL Parameter

Spark SQL magic does not have the same optional arguments that Trino does, and that is -c/--catalog where you provide a catalog for trino to use, and -m/--schema where you provide a schema for trino to use. This could be achieved in sparksql with the USE statement (e.g. USE catalog_name or USE catalog.schema_name), however, it would be useful to have these trino arguments as Spark SQL flags. This would allow users to only have to specify the table names, rather than the full namespace (i.e. catalog.schema.table).

[info-needed] Extension in VS code

Thank you for creating this extension. Do you plan to develop it into VS code?

sparksql magic: ValueError exception when a zero results returned.

Steps to reproduce:

• Create an SQL query with a filter that guarantees a results set with zero rows.
• Execute with %%sparksql -l 10 -c ...
• ValueError will be thrown.
  ValueError: Length of values (0) does not match length of index (3)

Json metadata format

Hey!

I'm using SparkMagics, so I need to manually generate the metadata JSON file to be used by the sql-language-server.

Could you provide an example of the JSON format?

Thanks in advance!

JupyterLab 4 compatibility question

Hi,
Is jupyterlab-sql-editor compatible and check with JupyterLab version 4 and jupyterlab-lsp version 5.0.0? Asking because i'm getting erros activating the plugins with the new versions.

@cccs-nik
Thanks

%trino SHOW CATALOGS adding LIMIT clause by default resulting in ERROR

Hi people of CyberCentreCanada.

Really loved the project, had a bit trouble installing but managed to use the Trino magics.

I noticed that in

https://github.com/CybercentreCanada/jupyterlab-sql-editor/blob/main/example/Trino.ipynb

there is a cell with '%trino SHOW CATALOGS' commented out. I realized that it wasn't working when trying out the command and that is because in your code there must be an automatic 'LIMIT n' on every query resulting in the error:

TrinoUserError: TrinoUserError(type=USER_ERROR, name=SYNTAX_ERROR, message="line 1:15: mismatched input 'limit'. Expecting: 'LIKE', <EOF>", query_id=20230504_184729_00795_e8dqv)

Managed to make it work by injecting comment dashes in the code and escaping the LIMIT clause.

%trino SHOW CATALOGS--

don't really know what you could do to fix that, but just wanted to show my workaround and provide feedback to this excelent project.

Cheers

Error - ModuleNotFoundError: No module named 'dbt.main'

Hello my friends.
I am facing the following problem after installing the extension in jupyterlab.

"ModuleNotFoundError: No module named 'dbt.main'"

My code:

import json
import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import ipywidgets as widgets
out = widgets.Output()
with out:
    spark = SparkSession.builder.getOrCreate()
%load_ext jupyterlab_sql_editor.ipython_magic.sparksql

Version of jupyterlab:
I'm running jupyterlab through a docker image: jupyter/all-spark-notebook

Can you help me please? I performed tests with other versions of the extension, I always get the same problem.

Request for tagged releases

I've been experimenting with the project and finding great utility in it. Thanks for creating it.
Would it be possible to tag the releases in github? This will help us find what code changes and dependencies have been made between versions easier.

Pre-configuration steps

I am trying this out for my use case and able to use it when I install it from the running notebook like

!pip3 install sparksql-magic
%load_ext sparksql_magic

but when I am following the contributing guide to pre-configure I am getting errors like ipython_magic module not found. I changed it to python_magic as well but the kernel start up logs always complain about the errors.

# pre-load the sparksql magic
c.InteractiveShellApp.extensions = [
    'ipython_magic.trino', 'ipython_magic.sparksql' // unable to find this module in pypi.org
    'python-magic.trino', 'python-magic.sparksql' //changed it to python-magic still getting module not found
]

//error during kernel startup, kernel launches fine but magic won't work.
[IPKernelApp] Loading IPython extension: storemagic
[IPKernelApp] Loading IPython extension: python_magic.sparksql
[IPKernelApp] WARNING | Error in loading extension: python_magic.sparksql
Check your config files in /app/.ipython/profile_custom
Traceback (most recent call last):
  File "/app/.local/lib/python3.7/site-packages/IPython/core/shellapp.py", line 301, in init_extensions
    self.shell.extension_manager.load_extension(ext)
  File "/app/.local/lib/python3.7/site-packages/IPython/core/extensions.py", line 80, in load_extension
    mod = import_module(module_str)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'python_magic'

--refresh all | local option should be a single option

When someone calls --refresh local and --refresh all was never called (so the JSON file does not exist), --refresh local will failed:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/.local/sparkdb.schema.json'
To fix this, the user needs to call --refresh all first to have the JSON file created. In my opinion, there should only be --refresh and the code should check if the JSON file exist. If it does, do a local refresh. If it doesn't, do all.

Autocomplete for lower case SQL keywords?

HI
Is there a feature to configure autocompletion for lower case SQL keywords too? I currently see it only for upper case SQL keywords

caching option should use view when possible

currently caching of the dataframe created by sparksql magic uses
df.cache()

This api does not let us name the cached data. Using the SQL version of this api it is possible to name the cache data (in the spark UI).

When the users creates a cached view
%%sparksql --output skip --view user_view_name --cache --eager

we could cache it using the SQL api as follows
df.createOrReplaceTempView("tmp_df_view"

spark.sql("CACHE TABLE user_view_name as select * from tmp_df_view")

spark.sql("DROP VIEW tmp_df_view")

Failed to load language server spec finder

Hello
Running into the following error when running jupyter lab. Also, had run pip install bokeh before this.

[W 2024-01-07 19:06:55.006 ServerApp] Failed to load language server spec finder `sparksql-language-server`: 
    'NVM_DIR'
[W 2024-01-07 19:06:55.061 ServerApp] Failed to load language server spec finder `trino-language-server`: 
    'NVM_DIR'

Auto completion does not work both on Windows 10 and ubuntu?

Autocomplete does not work on Windows 10 or ubuntu