jupyter / nbclient

A client library for executing notebooks. Formally nbconvert's ExecutePreprocessor

nbclient

NBClient lets you execute notebooks.

A client library for programmatic notebook execution, NBClient is a tool for running Jupyter Notebooks in different execution contexts, including the command line.

Interactive Demo

To demo NBClient interactively, click this Binder badge to start the demo:

Installation

In a terminal, run:

python3 -m pip install nbclient

Documentation

See ReadTheDocs for more in-depth details about the project and the API Reference.

Python Version Support

This library currently supports Python 3.6+ versions. As minor Python versions are officially sunset by the Python org, nbclient will similarly drop support in the future.

Origins

This library used to be part of the nbconvert project. NBClient extracted nbconvert's ExecutePreprocessorinto its own library for easier updating and importing by downstream libraries and applications.

Relationship to JupyterClient

NBClient and JupyterClient are distinct projects.

jupyter_client is a client library for the jupyter protocol. Specifically, jupyter_client provides the Python API for starting, managing and communicating with Jupyter kernels.

While, nbclient allows notebooks to be run in different execution contexts.

About the Jupyter Development Team

The Jupyter Development Team is the set of all contributors to the Jupyter project. This includes all of the Jupyter subprojects.

The core team that coordinates development on GitHub can be found here: https://github.com/jupyter/.

Our Copyright Policy

Jupyter uses a shared copyright model. Each contributor maintains copyright over their contributions to Jupyter. But, it is important to note that these contributions are typically only changes to the repositories. Thus, the Jupyter source code, in its entirety is not the copyright of any single person or institution. Instead, it is the collective copyright of the entire Jupyter Development Team. If individual contributors want to maintain a record of what changes/contributions they have specific copyright on, they should indicate their copyright in the commit message of the change, when they commit the change to one of the Jupyter repositories.

With this in mind, the following banner should be used in any source code file to indicate the copyright and license terms:

# Copyright (c) Jupyter Development Team.
# Distributed under the terms of the Modified BSD License.

nbclient's Issues

Traitlets 5.x and async_start_new_kernel bug incompat

The following method has a bug:

    async def async_start_new_kernel(self, **kwargs) -> None:
        """Creates a new kernel.

        Parameters
        ----------
        kwargs :
            Any options for ``self.kernel_manager_class.start_kernel()``. Because
            that defaults to AsyncKernelManager, this will likely include options
            accepted by ``AsyncKernelManager.start_kernel()``, which includes ``cwd``.
        """
        assert self.km is not None
        resource_path = self.resources.get('metadata', {}).get('path') or None
        if resource_path and 'cwd' not in kwargs:
            kwargs["cwd"] = resource_path

        if hasattr(self.km, 'ipykernel') and self.km.ipykernel and self.ipython_hist_file:
            extra_arguments +=  ['--HistoryManager.hist_file={}'.format(self.ipython_hist_file)]

        await ensure_async(self.km.start_kernel(extra_arguments=extra_arguments, **kwargs))

It mutates extra_arguments which appends HistoryManager.hist_file=... each time it is called; this is incompatible with traitlets 5.0 wich will fail if multiple values have been passed for the same option.

FutureWarning with jupyter_client 6.1.5

Since the release of jupyter_client 6.1.5 we obtain a FutureWarning on mne-python in our CIs:

FutureWarning: Method cleanup(connection_file=True) is deprecated, use cleanup_resources(restart=False).

More details

../../../miniconda/lib/python3.7/site-packages/nbclient/client.py:471: in async_setup_kernel
    await self._async_cleanup_kernel()
        cleanup_kc = True
        kwargs     = {}
        loop       = <_UnixSelectorEventLoop running=False closed=False debug=False>
        on_signal  = <function NotebookClient.async_setup_kernel.<locals>.on_signal at 0x7ff5361314d0>
        self       = <nbclient.client.NotebookClient object at 0x7ff538282790>
../../../miniconda/lib/python3.7/site-packages/nbclient/client.py:346: in _async_cleanup_kernel
    await ensure_async(self.km.cleanup())
        now        = False
        self       = <nbclient.client.NotebookClient object at 0x7ff538282790>
../../../miniconda/lib/python3.7/site-packages/jupyter_client/manager.py:358: in cleanup
    FutureWarning)
E   FutureWarning: Method cleanup(connection_file=True) is deprecated, use cleanup_resources(restart=False).

A standard for execution config metadata?

In one of our projects that uses nbclient, we need to allow for a per-notebook timeout limit (executablebooks/jupyter-cache#48). For example, people may want a "default" limit that is relatively low, but have "that one notebook" that they know takes way longer to execute, so they'd like to embed that information in the notebook in a way that can be utilized by downstream tools.

Is there a jupyter standard for adding execution-level metadata to a notebook? Can folks here recommend something reasonable? It would be good to have some standard practices so we don't start moving in separate directions unintentionally

Add ename and evalue attributes to CellExecutionError

This will make it easier to promote CellExecutionError to a native Python exception/error.

cc @MSeal

Backward incompatible change in patch release

It seems that we should have tagged a 0.5 instead of 0.4.2. #94 was a (much needed) backward incompatible change.

I suggest we tag a 0.5 right away and remove the 0.4.2 and 0.4.3 uploads as broken.

Add nbclient to conda-forge

@MSeal Seem like this would be helpful for packages that depend on nbclient.

info_msg contains no language_info

In https://readthedocs.org/projects/myst-nb/builds/11597535/,
I'm getting the error:

Traceback (most recent call last):
  File "/home/docs/checkouts/readthedocs.org/user_builds/myst-nb/envs/latest/lib/python3.7/site-packages/jupyter_cache/executors/basic.py", line 166, in execute
    record_timing=False,
  File "/home/docs/checkouts/readthedocs.org/user_builds/myst-nb/envs/latest/lib/python3.7/site-packages/nbclient/client.py", line 1077, in execute
    return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()
  File "/home/docs/checkouts/readthedocs.org/user_builds/myst-nb/envs/latest/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/home/docs/checkouts/readthedocs.org/user_builds/myst-nb/envs/latest/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/home/docs/.pyenv/versions/3.7.3/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/home/docs/checkouts/readthedocs.org/user_builds/myst-nb/envs/latest/lib/python3.7/site-packages/nbclient/client.py", line 532, in async_execute
    self.nb.metadata['language_info'] = info_msg['content']['language_info']
KeyError: 'language_info'

its really weird because, because I can't replicate it locally, or even in RTD PR Builds, its literally only when it comes to build latest after a merge/commit 😕

Run vs execute cell?

I was looking through the code and was confused by two methods that (from an api standpoint) seem they should be doing the same thing. One is execute_cell (

nbclient/nbclient/execute.py

Line 389 in 386e299

def execute_cell(self, cell, cell_index, store_history=True):

), and the other is run_cell (

nbclient/nbclient/execute.py

Line 504 in 386e299

def run_cell(self, cell, cell_index=0, store_history=False):

)

It seems like execute_cell is the preferred user-facing choice since it is documented more so. If so, maybe it makes sense to make run_cell a hidden function? Alternatively, we should document what is the difference

kernal info retrieval fails on small VMs

This is essentially re-opening #95, because #96 adds extra debugging, but it does not fix it.

Basically on nearly every RTD run that calls nbclient, it fails intermittently and is super frustrating 😞 e.g. https://readthedocs.org/projects/sphinx-book-theme/builds/11733784/.
This has happened for both v0.2 and v0.4

The VM they are using is Standard_D1_v2

See: https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series?toc=/azure/virtual-machines/linux/toc.json&bc=/azure/virtual-machines/linux/breadcrumb/toc.json#dv2-series

This may be related to #58

Make task for copying changelogs to readthedocs page

Either create an automated task to copy and convert the markdown changelogs to the readthedocs /docs path or just move the changelog to rst and place it there (I dislike writing rst by hand).

The use of nest_asyncio makes `nbconvert --to webpdf` hang when used with `--execute`

This comes with an issue in pyppeteer: pyppeteer/pyppeteer#99

How could we avoid using nest_asyncio for nbconvert --execute?

Add a small CLI?

How do folks feel about a lightweight CLI that lets people execute notebooks from the command-line? e.g., something that would replace

jupyter nbconvert --to notebook --execute mynotebook.ipynb

Something like

jupyter execute mynotebook.ipynb

or

nbclient execute mynotebook.ipynb

?

Alternatively, we could recommend that people use papermill if they wish to execute from the command line? (https://papermill.readthedocs.io/en/latest/usage-execute.html)

Not getting a CellExecutionError raised with %run commands

I follow the practice of creating a master notebook that runs all the notebooks in my pipeline via the %run magic command. When I execute that notebook with nbclient, it fails to raise errors when they happen in the sub-notebooks. This is, for obvious reasons, not good. Is there a way to address this?

Tests fail on 1-core machine

While working on reproducible builds for openSUSE, I found that our python-nbclient package fails tests on a 1-core VM

Is this a bug or expected behaviour?
jmcarpenter2/swifter#102 solved a similar issue by skipping such tests when not enough cores were available.

to reproduce on openSUSE or Debian:

osc co openSUSE:Factory/python-nbclient && cd $_
osc build --vm-type=kvm -j1 --noservice standard

The issue can also be triggered via taskset 1 pytest-3.8

example failure log:

 nbclient/tests/test_client.py:270: 
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
 expected = {'cells': [{'cell_type': 'code', 'execution_count': 1, 'metadata': {'tags': ['raises-exception']}, 'outputs': [{'ename...utput_type': 'stream', 'text': 'ok\n'}], 'source': "print('ok')"}], 'metadata': {}, 'nbformat': 4, 'nbformat_minor': 1}
 actual = {'cells': [{'cell_type': 'code', 'metadata': {'tags': ['raises-exception'], 'execution': {'iopub.status.busy': '2020-0...ents_lexer': 'ipython3', 'nbconvert_exporter': 'python', 'file_extension': '.py'}}, 'nbformat': 4, 'nbformat_minor': 1}
 
     def assert_notebooks_equal(expected, actual):
         expected_cells = expected['cells']
         actual_cells = actual['cells']
         assert len(expected_cells) == len(actual_cells)
     
         for expected_cell, actual_cell in zip(expected_cells, actual_cells):
             # Uncomment these to help debug test failures better
             # from pprint import pprint
             # pprint(expected_cell)
             # pprint(actual_cell)
             expected_outputs = expected_cell.get('outputs', [])
             actual_outputs = actual_cell.get('outputs', [])
             normalized_expected_outputs = list(map(normalize_output, expected_outputs))
             normalized_actual_outputs = list(map(normalize_output, actual_outputs))
 >           assert normalized_expected_outputs == normalized_actual_outputs
 E           AssertionError: assert [{'name': 'st...ext': 'ok\n'}] == []
 E             Left contains one more item: {'name': 'stdout', 'output_type': 'stream', 'text': 'ok\n'}
 E             Full diff:
 E             - []
 E             + [{'name': 'stdout', 'output_type': 'stream', 'text': 'ok\n'}]
 
 nbclient/tests/test_client.py:228: AssertionError
 ____________________ TestExecute.test_cleanup_kernel_client ____________________
 
 self = <nbclient.tests.test_client.TestExecute testMethod=test_cleanup_kernel_client>
 
     def test_cleanup_kernel_client(self):
         filename = os.path.join(current_dir, 'files', 'HelloWorld.ipynb')
     
         with io.open(filename) as f:
             input_nb = nbformat.read(f, 4)
     
         executor = NotebookClient(
             input_nb,
             resources=self.build_resources(),
         )
     
         executor.execute()
         # we asked to cleanup the kernel client (default is True)
         assert executor.kc is None
     
 >       executor.execute(cleanup_kc=False)

Intermittent IOPub timeout problems on windows

When running windows CI, I observe sporadic execution failures with a warning

WARNING | Timeout waiting for IOPub output.

Related reports: mwouts/jupytext#489 jupyter/jupyter-sphinx#157
There was a WIP PR against nbconvert that seemingly should fix this (jupyter/nbconvert#1183), but it was seemingly not finished nor taken over.

Why isn't this project called "nbexecute"?

I was wondering what nbclient is supposed to do except executing ... and I asked there: #4 (comment)

This was answered by @MSeal in #4 (comment):

For now the intention is to have a very narrow scoped notebook executor that has few dependencies, like jupyter_client does for interacting with kernels. This library doesn't handle output IO and does one thing well. It'll likely get an async execution pattern, but beyond that I don't see major responsibilities being added to the core the library for now.

So if it is only ever supposed to "execute", why call it by a totally un-intuitive and confusing name?

kernel returning None in metadata can cause nbclient to stall

This is tracking down voila-dashboards/voila#682 and Calysto/metakernel#212

Voila will stall if metakernel returns a None in metadata. This has been fixed in metakernell#212, but nbclient should not stall if the kernel returns invalid data as this can create a ddos attack.

The issue seems to be that _update_display_id updates outputs[output_idx]['metadata'] to None, then _async_poll_for_reply tries to update metadata. If there is a error in that loop the try-catch block will catch the exception and the function will stay in the loop. This causes a stall as the kernel is still alive.

Should multi-notebook execution (and a few other features) be in-scope for this package?

We're working on another project that will likely involve some kind of multi-notebook output cacheing and execution. Things like:

• A class that can parallelize the execution of notebooks
• A way to check whether a notebook has become "stale" and should be re-executed to make sure outputs match the code cells or content
• A way to generate information about the execution status (stale, fresh, error states, etc) of multiple notebooks

I am trying to decide whether nbclient is the right home for these kinds of features, or if instead we should plan on depending on nbclient, but doing these features as part of another package. I'm happy w/ either approach, but curious what others think.

0.4.0 alpha

Would anyone be opposed to a prerelease of 0.4.0?

We would like to test the port of the Output widget support in Voilà, and that would simplify CI.

nbclient.client.setup_kernel context manager does not shutdown kernel on exit

Discovered while investigating a suspected memory leak in the downstream papermill library.

I had to monkey patch this to insert self.km.shutdown_kernel() to make it work.

confusing intro text

The text on https://nbclient.readthedocs.io/en/latest: "Similar in nature to jupyter_client, as the jupyter_client is to the jupyter protocol nbclient is to notebooks allowing for execution contexts to be run." is fairly hard to parse. I was going to put in a PR to make it more clear, but I couldn't quite parse it well enough to figure out how to do so.

Can you pass environmental variables to the kernel?

I was thinking of this enabling the running of certain setup code something like:

import os

if "JUPYTER_BOOK" in os.environ:
  %config InlineBackend.figure_format = "retina"
  plt.rcParams["axes.spines.right"] = False
  plt.rcParams["axes.spines.top"] = False

(related to executablebooks/meta#151 (reply in thread))

Why is the kernel client class pinned to AsyncKernelClient?

Hello - great project folks!

I'm trying to bring my own kernel manager class for running a notebook through papermill which needs a specific kernel client class. However, nbclient is forcing the use of AsyncKernelClient despite my kernel manager specifying that a different client class be used. I suspect this is to trigger the use of AsyncKernelClient even when the synchronous jupyter_client KernelManager class is used but would like to get a better understanding.

If this is the reason, it seems like this forced override should be conveyed via a configuration option so that custom kernel client classes can be used. In reading through #10, I didn't find any discussion of this particular line of code - which I also found surprising since the kernel manager class is already responsible for determining its corresponding kernel client class.

I'm happy to submit a PR that introduces a boolean to trigger this override - or something to that effect - so long as there's an ability to use the desired kernel client class specified by the kernel manager. Thanks.

Rename start_kernel_manager into create_kernel_manager

The former makes it look like it is starting a kernel or something like this, which it is just instantiating the kernel manager class.

Having trouble running nbclient 0.3.0

Apologies if I'm doing something dumb. I've just started trying to play around with 0.3.0, and I'm running into some issues.

Context: I'm trying to get enterprise gateway's RemoteKernelManager to work on its own as a KernelManager (jupyter-server/enterprise_gateway#803). This is a sync KernelManager, so I'm guessing that's why it's not working as expected.

import nbformat
from nbclient import NotebookClient
from enterprise_gateway.services.kernels.remotemanager import RemoteKernelManager

with open("/home/ish/notebooks/Untitled.ipynb") as fp:
    test_notebook = nbformat.read(fp, as_version=4)

client = NotebookClient(nb=test_notebook, kernel_manager_class=RemoteKernelManager)
client.execute()

The above code works as expected wth 0.2.0, but there's a lot of strange behaviour in 0.3.0.

It also works with 0.1.0, but doesn't shutdown. I don't see shutdown_kernel called anywhere besides on client startup failure, so I guess that's to be expected? Although the docstring does say the kernel gets shut down.

• When I run the above, the kernel executes as expected, but is never shut down. Still looking into why this is the case (is this intended? possibly an issue on my end)
• When I run the above, but with reset_kc=True, I run into a lot of issues, first of which is that in execute, self.kc has not been initialized from the class before self._async_cleanup_kernel is called, so that fails and while it's handling that failure, it fails again on self.km.cleanup() since self.km doesn't exist.

I am using an unusual KernelManager, but I don't think that's the cause of these issues.

Couple other things I noticed; the changelog link is broken, and there's a typo in the docstring for NotebookClient (Kernel spelled Kernerl)

Async not awaited in some tests.

Looks like it's mostly benign around test mocking not being updated to await always, but we probably want to remove the warnings before releasing in case there's a bug associated? Thoughts @davidbrochart

=============================== warnings summary ===============================
nbclient/tests/test_client.py::TestRunCell::test_deadline_iopub
  /usr/lib/python3.6/unittest/mock.py:684: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
    return object.__setattr__(self, name, value)

nbclient/tests/test_client.py::TestRunCell::test_display_data_message
nbclient/tests/test_client.py::TestRunCell::test_unknown_comm_message
  /usr/lib/python3.6/unittest/mock.py:1898: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
    self.name = name

nbclient/tests/test_client.py::TestRunCell::test_eventual_deadline_iopub
  /home/mseal/Workspace/nbclient/nbclient/tests/test_client.py:719: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
    message_mock.side_effect = message_seq(list(message_mock.side_effect)[:-1])

nbclient/tests/test_client.py::TestRunCell::test_eventual_deadline_iopub
  /home/mseal/.py3local/lib/python3.6/site-packages/traitlets/traitlets.py:510: RuntimeWarning: coroutine 'make_async.<locals>._' was never awaited
    if meth_name in cls.__dict__:

Impossible to execute notebooks containing the %gui qt magic?

When executing a notebook containing a %gui qt magic, nbclient fails with the error:

AttributeError: 'IPythonKernel' object has no attribute 'app'

[IPKernelApp] ERROR | Exception in control handler:
Traceback (most recent call last):
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/ipykernel/kernelbase.py", line 201, in dispatch_control
    yield gen.maybe_future(handler(self.control_stream, idents, msg))
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/tornado/gen.py", line 735, in run
    value = future.result()
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/tornado/gen.py", line 209, in wrapper
    yielded = next(result)
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/ipykernel/kernelbase.py", line 683, in shutdown_request
    content = yield gen.maybe_future(self.do_shutdown(parent['content']['restart']))
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/ipykernel/ipkernel.py", line 450, in do_shutdown
    self.shell.exit_now = True
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/traitlets/traitlets.py", line 585, in __set__
    self.set(obj, value)
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/traitlets/traitlets.py", line 574, in set
    obj._notify_trait(self.name, old_value, new_value)
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/traitlets/traitlets.py", line 1134, in _notify_trait
    self.notify_change(Bunch(
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/traitlets/traitlets.py", line 1176, in notify_change
    c(change)
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/ipykernel/zmqshell.py", line 476, in _update_exit_now
    exit_hook(self.kernel)
  File "/Users/jni/conda/envs/all/lib/python3.8/site-packages/ipykernel/eventloops.py", line 136, in loop_qt_exit
    kernel.app.exit()
AttributeError: 'IPythonKernel' object has no attribute 'app'
Out[6]:
{'cells': [{'cell_type': 'code',
   'execution_count': 1,
   'metadata': {'execution': {'iopub.status.busy': '2020-08-10T13:24:48.262059Z',
     'iopub.execute_input': '2020-08-10T13:24:48.262847Z',
     'iopub.status.idle': '2020-08-10T13:24:48.354775Z',
     'shell.execute_reply': '2020-08-10T13:24:48.356131Z'}},
   'outputs': [],
   'source': '%gui qt\n%matplotlib qt'},
  {'cell_type': 'code',
   'execution_count': 2,
   'metadata': {'execution': {'iopub.status.busy': '2020-08-10T13:24:48.359983Z',
     'iopub.execute_input': '2020-08-10T13:24:48.360516Z',
     'iopub.status.idle': '2020-08-10T13:24:48.537911Z',
     'shell.execute_reply': '2020-08-10T13:24:48.538351Z'}},
   'outputs': [{'output_type': 'execute_result',
     'metadata': {},
     'data': {'text/plain': '<matplotlib.collections.PathCollection at 0x7f837a09feb0>'},
     'execution_count': 2}],
   'source': 'import matplotlib.pyplot as plt\nimport numpy as np\n\n\nplt.scatter(*np.random.random((2, 10)))'}],
 'metadata': {'kernelspec': {'display_name': 'Python 3',
   'language': 'python',
   'name': 'python3'},
  'language_info': {'name': 'python',
   'version': '3.8.2',
   'mimetype': 'text/x-python',
   'codemirror_mode': {'name': 'ipython', 'version': 3},
   'pygments_lexer': 'ipython3',
   'nbconvert_exporter': 'python',
   'file_extension': '.py'}},
 'nbformat': 4,
 'nbformat_minor': 4}

I've provided a minimal notebook in this gist.

In turn, the nbclient code to run it (basically lifted from the getting started guide):

import nbformat
from nbclient import NotebookClient
with open('Untitled.ipynb') as f:
    nb = nbformat.read(f, as_version=4)

client = NotebookClient(nb, timeout=15, kernel_name='python3')
client.execute()

xref: jupyter-book/jupyter-book#840

"class-instead-of-function" anti-pattern in Executor class

I've already mentioned this in #12 (comment), but I think this deserves its own issue.

During the move from nbconvert to nbclient, the Executor class evolved into something that's not really a meaningful class anymore. Instead, it is something that takes a few input parameters and produces an output. In other words, it turned into a function disguised as a class.

This is clearly visible by its usage here:

nbclient/nbclient/execute.py

Line 721 in ad2eba7

return Executor(nb=nb, resources=resources, km=km, **kwargs).execute()

This was somewhat better in nbconvert (see https://github.com/jupyter/nbconvert/blob/8c9bffc8deb65ced99b20684ddd148fd885ad9e8/nbconvert/preprocessors/execute.py#L765-L766):

    ep = ExecutePreprocessor(**kwargs)
    return ep.preprocess(nb, resources, km=km)[0]

I think there are two possible ways to improve this, I don't know which one is better in this case:

1. Get rid of the Executor class and implement everything in the stand-alone execute() function. This would of course become a huge function, so probably some parts of it should be factored out in separate (probably internal) functions and classes.

2. Make the Executor great again. An Executor class should be instantiated with some set of options and it should provide an execute() method that can be repeatedly (and ideally concurrently) be used to execute multiple notebooks with those given options.

Issue with typing.AscyncGenerator for python 3.6.0

Hi there,

I upgraded to nbconvert 6.0.7 from from 5.6.1 recently, so as part of that I installed nbclient 0.5.1.

python 3.6.0
typing 3.7.4.3
notebook 6.1.4

I'm running python 3.6.0 on this machine. After the upgraded I started getting these errors when I tried to start a notebook (500, internal error page):

#....
File "h:\anaconda3\lib\site-packages\nbclient\client.py", line447, in NotebookClient
  async def async_setup_kernel(self, **kwargs) -> t.AsyncGenerator:
AttributeError: module 'typing' has no attribute 'AsyncGenerator'

If I revert back to nbconvert 5.6.1 life is back to normal again, notebooks can be opened.

On another machine with python 3.7, I do not see this issue. This might be specifically related to python 3.6.0.

Unfortunately I couldn't upgrade the machine to 3.7 easily... Any idea how / why this was happening to 3.6.0?

Thanks.

Master test failure

Thee tests on master appear to be failing on Windows + Python 3.6 over OSError: [WinError 6] The handle is invalid.

I re-started the tests to see if this occurs all the time, because the PR that was merged last was passing.

Function hooks when executing cells / notebooks?

I'm trying to find a way to achieve scrapbook-like functionality but without asking users to write new code. Instead, I'd like them to be able to tag cells with metadata (either tags like glue_varname, or other metadata) and to programatically embed extra outputs in the cell depending on that metadata.

I think to do this with nbclient, we could allow either a notebook-level or cell-level hook that could be run alongside the call to execute(). Basically, a function that runs in the context of the kernel that is run with the notebook. I'd have a preference for the ability to do this in each cell, but notebook-level could work too.

It seems like this could potentially be useful for other kinds of usecases too, so I wonder what folks think about supporting something like this. Alternatively, can anybody think of a way to support the usecase I described at the top?

Any interest in the sphinx-book-theme for docs?

We've been working on a simple sphinx theme for the jupyter-book rewrite that I've also found useful for other documentation sites as well (sphinx-book-theme.readthedocs.io)

I find the Alabaster theme to be a little bit cluttered and hard to parse sometimes...would folks be interested in a PR that updates the docs theme for nbclient to the sphinx-book-theme? Here's an example of what it looks like by just swapping the theme name in conf.py:

Add execution timing to match latest nbformat version spec

We need to add support for jupyter/nbformat#144 to be spec compatible with 4.4

Discuss or document how to include widget output

Over in executablebooks/jupyter-cache#31 @chrisjsewell and I ran into a bunch of confusing issues where executing notebooks results in widget state not making it into the output notebook. We've tried a bunch of stuff around "trusting" the notebook etc but can't seem to figure it out.

In the docs we allude to this:

If you can’t view widget results after execution, you may need to select Trust Notebook under the File menu.

But is there a place to programmatically "trust" a notebook so one can do it just before running nbclient?

I imagine that others that use nbclient might run into the same issues, so perhaps this issue is worth documenting a bit more.

Python 3.6 support (nbconvert)

Hi Team,

Our code is using this library & it was working fine & from today we are getting below error:

    from nbconvert import HTMLExporter
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbconvert/__init__.py", line 4, in <module>
    from .exporters import *
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbconvert/exporters/__init__.py", line 4, in <module>
    from .slides import SlidesExporter
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbconvert/exporters/slides.py", line 12, in <module>
    from ..preprocessors.base import Preprocessor
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbconvert/preprocessors/__init__.py", line 10, in <module>
    from .execute import ExecutePreprocessor
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbconvert/preprocessors/execute.py", line 6, in <module>
    from nbclient import NotebookClient, execute as _execute
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbclient/__init__.py", line 3, in <module>
    from .client import NotebookClient, execute  # noqa: F401
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbclient/client.py", line 39, in <module>
    class NotebookClient(LoggingConfigurable):
  File "/home/mosaic-ai/.pyenv/versions/3.6.0/lib/python3.6/site-packages/nbclient/client.py", line 442, in NotebookClient
    async def async_setup_kernel(self, **kwargs) -> t.AsyncGenerator:
AttributeError: module 'typing' has no attribute 'AsyncGenerator'

Initial 0.1.0 Release Plan

• Point readthedocs to this repo
• Setup circileci to look at this repo
• Automate a changelog copy to docs page?
• Do the first release (0.1.0?)
• Update nbconvert 6.0 alpha to import this package
• Update papermill 2.0 (master) to import this package

async_execute always creates a new kernel client

Should we allow the possibility to pass an external kernel client?

Allow the kernel manager to shut down gracefully in _async_cleanup_kernel

Referencing this code here:
https://github.com/jupyter/nbclient/blob/0.3.0/nbclient/client.py#L331-L348

client.shutdown only sends the shutdown message to the kernel. When it completes its execution, it's not guaranteed that the kernel is shutdown. And immediately after that, the kernel manager is force-killing the kernel, using SIGKILL. (I haven't experienced this before, but couldn't shutdown hooks be affected by this?)

Since I don't know much about the decision to implement this function in this way, I'd like your input on potentially changing it to try a graceful shutdown using the kernelmanager with a timeout or something before forcefully terminating.

Errors after upgrading to 0.3.0

Some users following a pattern similar to the one described in #48 are getting new errors after upgrading to 0.3. When they downgrade to 0.2, the errors are gone. Here's a sample traceback.

Traceback (most recent call last):
  File "update.py", line 154, in <module>
    cli()
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "update.py", line 85, in process
    _execute_notebook(
  File "update.py", line 29, in _execute_notebook
    client.execute()
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/nbclient/util.py", line 37, in wrapped
    result = loop.run_until_complete(coro(self, *args, **kwargs))
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 612, in run_until_complete
    return future.result()
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/nbclient/client.py", line 471, in async_execute
    async with self.async_setup_kernel(**kwargs):
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/nbclient/client.py", line 440, in async_setup_kernel
    await self.async_start_new_kernel_client(**kwargs)
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/nbclient/client.py", line 393, in async_start_new_kernel_client
    await ensure_async(self.kc.start_channels())
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/jupyter_client/client.py", line 116, in start_channels
    self.hb_channel.start()
  File "/Users/apesce/.local/share/virtualenvs/coronavirus-tracker-dljAgG8F/lib/python3.8/site-packages/jupyter_client/asynchronous/client.py", line 97, in hb_channel
    loop = asyncio.new_event_loop()
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 758, in new_event_loop
    return get_event_loop_policy().new_event_loop()
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 656, in new_event_loop
    return self._loop_factory()
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/unix_events.py", line 54, in __init__
    super().__init__(selector)
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/selector_events.py", line 61, in __init__
    self._make_self_pipe()
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/selector_events.py", line 108, in _make_self_pipe
    self._ssock, self._csock = socket.socketpair()
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/socket.py", line 571, in socketpair
    a, b = _socket.socketpair(family, type, proto)
OSError: [Errno 24] Too many open files
Exception ignored in: <function BaseEventLoop.__del__ at 0x1065fec10>
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 652, in __del__
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/unix_events.py", line 58, in close
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/selector_events.py", line 92, in close
  File "/usr/local/Cellar/[email protected]/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/selector_events.py", line 99, in _close_self_pipe
AttributeError: '_UnixSelectorEventLoop' object has no attribute '_ssock'
make[1]: *** [process] Error 1
make: *** [update] Error 2
(coronavirus-tracker) apesce@anthonys-mbp coronavirus-tracker % [IPKernelApp] WARNING | Parent appears to have exited, shutting down.
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.
[IPKernelApp] WARNING | Parent appears to have exited, shutting down.

Issue running nbclient with a gateway url

MappingKernelManager cannot be used as kernel_manager_class and I only just realized it didn't inherit from KernelManager. Looking at the source, it seems they're very different things, but that's confusing given the names.

Since GatewayKernelManager doesn't inherit from KernelManager, it doesn't seem like there's a way to get this working.

Am I doing something dumb here?

Nbclient waits indefinitely on killed subprocesses

We use nbclient with papermill to run notebooks in docker containers. If a notebook exceed the allocated memory (of the container) only the notebook subprocess seems to be killed (or dies), this seems to be not detected by nbclient, resulting in an infinite running process.

import nbformat
from nbclient import NotebookClient

print("Start")
with open("test_notebook.ipynb") as f:
    nb = nbformat.read(f, as_version=4)

client = NotebookClient(nb, kernel_name="python3")
client.execute()

print("Done")

This issue can be reproduced in two ways, with the script above:

• Running a python debuger, setting a breakpoint here https://github.com/jupyter/nbclient/blob/master/nbclient/client.py#L521 and killing the subprocess by hand using the system monitor, and then continuing execution. Note: This can create the same type of issue (stuck in an infinite loop), but the solution might not completely overlap with the issue we are experiencing.
• Or using the following steps with docker. :

First ensure that "cgroup swap limit capabilities" are enabled (if you have a lot of swap space). I used Ubuntu. See here for more info:
https://docs.docker.com/engine/install/linux-postinstall/#your-kernel-does-not-support-cgroup-swap-limit-capabilities

Create the following files

Dockerfile:

FROM python:3
WORKDIR /usr/src/app
RUN pip install numpy ipykernel nbclient
COPY . .
CMD [ "python", "./test.py" ]

test_notebook.ipynb:

import numpy as np

test = np.random.random((110, 10000000))

Build and run docker container (this will result in an infinite running process):

docker build -t my-python-app .
docker run -it --memory-swap 1000m --memory 1000m --rm --name my-running-app my-python-app

Run docker container without any issues:

docker run -it --rm --name my-running-app my-python-app

The issue may be caused by a missing timeout on the following line, but adding a timeout does not work for me:

nbclient/nbclient/client.py

Line 627 in 4314a44

msg = await ensure_async(self.kc.iopub_channel.get_msg(timeout=None))

broken markdown in README

NBClient lets you:

• execute notebooks

instead of

NBClient lets you:

**execute** notebooks

Exception in async_setup_kernel

I am getting the following error in nbclient when running voila 0.2 alpha with the classic template

  File "voila/handler.py", line 126, in _jinja_notebook_execute
    result = await self.executor.async_execute()
  File "nbclient/client.py", line 520, in async_execute
    async with self.async_setup_kernel(**kwargs):
  File "async_generator/_util.py", line 34, in __aenter__
    return await self._agen.asend(None)
  File "nbclient/client.py", line 478, in async_setup_kernel
    if not self.km.has_kernel:
AttributeError: 'AsyncMappingKernelManager' object has no attribute 'has_kernel'

CHANGELOG is a 404

Release

When do you plan to make a release? We need one in order to have async execution in Voila (voila-dashboards/voila#541).
Also, I saw that nbclient was released in PyPI (https://pypi.org/project/nbclient/#history) but not in the GitHub repo.

Type annotations

We're on python 3 only, let's make use of type annotations on our functions.

Widget Output objects save wrong display content

See jupyter/nbconvert#923 for root-cause analysis and proposal for solution of the issue for the code base when it was in nbconvert.

Asyncio Python 3.8

See this related issue: nteract/papermill#515, but essentially the async implementation on windows 3.8 uses a different implementation that doesn't include all the same functions. This means pyzqm barfs on loop actions. We likely need to patch nbclient in a similar way? I know this should only really be set on entrypoint but from testing setting it multiple times works just fine and the reality is there is no other event loop policy right now that we'd be worried about overwriting.

Document allowing errors with cell tags

Maybe I'm missing it, but I couldn't find the ability to mark cells as "raises errors" with tags:

nbclient/nbclient/client.py

Line 543 in 507876f

cell_allows_errors = self.allow_errors or "raises-exception" in cell.metadata.get(

if this is a feature we want to support (I think it'd be useful) we should document it!

1.0 Release Goals

I wanted to open up a discussion for what we consider needed for a 1.0 release. I think the core code is shaping up to be solid and getting tested well in a number of environments. Is there anything folks see as "should have" or "must have" for 1.0?

My starting note would be to make sure we have an interface pass for flags, old control planes that we maybe don't want to support because of how-it-was in nbconvert. Essential do a check if any config options don't make sense or should be added / removed.

Additionally I think making interface frictions like what came up here nteract/papermill#490 less annoying and handled more automatically.