jupyter / dashboards_server

[RETIRED] Server that runs and renders Jupyter notebooks as interactive dashboards

Jupyter

Read this in other languages: English, Español, Português, Français

Jupyter metapackage for installation and documents

Documentation structure

This documentation uses the Sphinx documentation engine.

The documentation is located in the docs/source folder. When you build the documentation, it will be placed in the docs/build folder. It is written in a combination of reStructuredText and MyST Markdown.

Build the documentation locally

There are a few ways to build the documentation; see below for instructions:

Build the documentation automatically with nox

The easiest way to build the documentation locally is by using the nox command line tool. This tool makes it easy to automate commands in a repository, and we have included a docs command to quickly install the dependencies and build the documentation.

To build and preview the site locally, follow these steps:

1. Clone this repository.

   $ git clone https://github.com/jupyter/jupyter
   $ cd jupyter

2. Install nox

   $ pip install nox

3. Run the docs command

   $ nox -s docs

This will install the needed dependencies in a virtual environment using pip. It will then place the documentation in the docs/build/html folder. You may explore these HTML files in order to preview the site.

Create a live server to automatically preview changes

There is another nox command that will do the above, and also create a live server that watches your source files for changes, and auto-builds the website any time a change is made.

To start this live server, use the following nox command:

$ nox -s docs-live

When the build is finished, go to the URL that is displayed. It should show a live preview of your documentation.

To stop serving the website, press Ctrl-C in your terminal

Build the documentation manually

To build the documentation manually, follow these steps:

First, install the miniconda Python distribution.

Next, navigate to the /docs directory and create a conda environment:

conda env create -f environment.yml

Activate the environment:

source activate jupyter_docs

Build the docs using Sphinx with the following commands:

make clean
make html

The docs will be built in build/html. They can be viewed by opening build/html/index.html or starting an HTTP server and navigating to 0.0.0.0:8000 in your web browser.

python3 -m http.server

Releasing the jupyter metapackage

Anyone with push access to this repo can make a release of the Jupyter metapackage (this happens very rarely). We use tbump to publish releases.

tbump updates version numbers and publishes the git tag of the version. Our GitHub Actions then build the releases and publish them to PyPI.

The steps involved:

1. Install tbump: pip install tbump
2. Tag and publish the release tbump $NEW_VERSION.

That's it!

dashboards_server's Issues

Heading and font size stylings differ between notebook dash preview and dashboard server

Use https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_layout_basic.ipynb as a reference. In dashboard server, the h1 is smaller than the h2, for example. Headings not bolded the same way. And so on. Makes layout difficult to get right.

vs

Allow uploading of zip containing notebook with resources

The ability to upload a notebook to the dashboards server does not allow for resources to go with it, such as widget files and data. Allow a zip file to be uploaded and its resources unpacked and served without clobbering the resources of other notebooks.

---

Structure: [updated 2016-02-26]

<notebook_name>/
    index.ipynb
    urth_components/
        ...
    stylesheet.css
    image.jpg

---

Client/Server Tasks:

• Support plain and bundled dashboards.
  • Plain & bundled should look same in UI.
  • Don't show ipynb extension.
• Support GET of bundled resources (everything but the notebook itself).
• Look for index.ipynb and always render for that dashboard for the dir, whether inside or outside of a "bundled" dashboard.
  • Look for index instead of __notebook__
• Load declwidgets from within bundled dashboard
• Create example installable through Makefile. Should test declwidgets, loading of local resources (CSS, images, etc).
• Support POST of bundled dashboards into data dir
• Create tests
  • Test loading index.ipynb (with/without extension). We should not get full contents back.
  • etc
• Update README to describe use of index.ipynb in a dir and local resources.

Get a demo with widgets and plots working

Go beyond the basic simple / web3. See what happens.

jupyter-js-output-area relocated to jupyter-js-notebook

npm WARN deprecated [email protected]: This code was merged into the jupyter-js-notebook package.

Support ipywidget based interactive elements

The impl in #1 will not be able to support ipywidgets or declarative widgets yet:

WRT ipywidgets support, there isn't support for jupyter-js-services yet. I'll probably end up moving to the jupyter-js-services comm API and shimming the current API (notebook compatibility), for ipywidgets 5.0 (next release) or 6.0 (depends on how much time I can allocate to 5.0).

From jupyter/dashboards#105 (comment)

Once the ipywidgets support for jupyter-js-services lands, the app here can start to take advantage of it.

Hide stderr / exception tracebacks in dashboard view

Port of the styling added to jupyter-incubator/dashboards and jupyter-incubator/dashboards_bundlers recently.

Cells overlapping again

No console errors. Looks like something else broke gridstack. Confirmed that lodash is pinned. Even tried setting it to 3.10.1 exactly.

test_busy_indicator.ipynb text size difference, cut off

Text cutoff in dashboard server, but fine in notebook https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_busy_indicator.ipynb.

Dashboard backend can proxy to KGs that don't run at root of domain

Kernel gateway supports a base_url parameter that allows it to run under non-root paths. This is necessary when it's deployed using something like tmpnb which runs multiple instances of the KG.

Make sure the dashboard server can:

• Proxy to a single kernel gateway that is running under a custom base_url
• [ ] Respects the URL returned by tmpnb's /api/spawn (see #11)

Use Node streams instead of overriding `socket.emit`?

We have an ugly bit of code in api.js setupWSProxy() where we override socket.emit in order to add in the actual notebook code to be executed. It would be cleaner to use Node streams, if possible.

The http-proxy library does the proxying of websocket connections with the following code in ws-incoming.js:

proxySocket.pipe(socket).pipe(proxySocket);

I believe the socket variable above is what we pass in to the proxy lib in api.js:

proxy.ws(req, socket, head);

(not 100% sure about that, though).

If my assumptions above are correct, we should then be able to create our own Duplex Stream implementation that will change the incoming WS messages. In the end, we want the equivalent of:

proxySocket.pipe(wsRewriter).pipe(socket).pipe(proxySocket);

Since we don't care about the return WS msgs, we only need to insert our object (wsRewriter) on the incoming side.

Style navbar

We currently have a navbar with default bootstrap styling. We could style it to integrate better into the page and emphasize the page content.

Fix progress bar (and other styling)

Fix styling issues:

• Busy indicator appears at bottom of page, doesn't animate and doesn't disappear.
• Font sizes are different than dashboard in Notebook.
• Stream message outputs get appended as new elements in output area
  • When phosphor-widget is updated to 1.0.0-rc1 or later, re-commit 94d734d.
• Output area overlaps widget area, but only when running from Docker container (not from localhost).

NodeJS app should send kernel shutdown when client websocket disconnects

Most likely fix, call the jupyter-js-services shutdown method for the disconnecting websocket instance here: https://github.com/jupyter-incubator/dashboards_nodejs_app/blob/5ace686d7efd80eb1f70f3f8fc658c5a72504bf8/routes/api.js#L154 and in other errors cases too.

Prevent re-running of code cells

Current implementation prevents arbitrary code from the browser to reach the kernel. However, the browser can instruct the proxy to resend one of the code cells from a notebook multiple times. A bad actor might take advantage of this to mess with state on the backend if the notebook code is not written to account for multiple runs (e.g., incrementing a counter too many times).

Still, this is not a serious bug: the code already has to run once on the first load of the dashboard.

Update to jupyter-js-services 0.5

(We're currently at 0.3.x)

Major change is that commOpened signal is gone, replaced with registerCommTarget().

---

Tasks:

• Replace use of commOpened with registerCommTarget().
• Fix display_data bug with jupyter-js-services (jupyter/services/pull/94)
• Update ipywidgets to use latest jupyter-js-services (jupyter-widgets/ipywidgets/issues/381)
• Update declarativewidgets to latest jupyter-js-services.
• Update declarativewidgets to latest ipywidgets (jupyter/declarativewidgets#157 (comment)).
• Don't build jquery-ui into ipywidgets, since we also use it and this leads to code duplication and bloat.

Allow multiple connections to view currently running kernel

For a deployed dynamic dashboard to be usable, people need to load an existing dashboard and immediately see its current state. For dashboards that are long-running (streaming) or that take time to run (Spark), it is not practical to re-run the code in a new kernel each time the page is loaded.

We should allow new connections to an already running kernel and properly build the dashboard with the required cell outputs and interactivity without having to re-run code.

Support an optional Authorization: token <value> check for POST /notebooks

Roughly here:

https://github.com/jupyter-incubator/dashboards_nodejs_app/blob/master/routes/index.js#L63

jupyter/dashboards_bundlers#12 will be the first client to use it.

Doc and automation

• All params are nconf
• Expose IP interface as setting (default to 127.0.0.1)
• All params are documented in config.json and shown in make help
• Complete README sections
• make test within docker container
• Make cert targets real targets in makefile

Read Urth.* defaults from config

Currently hardcoded in dashboard.handlebars. See TODO at the bottom of that file.

POST handler for adding a notebook to the running instance

It would be handy if the node app supported POST /notebooks to add / overwrite an existing notebook with the same with a new one for quick and dirty deployment.

https://github.com/jupyter-incubator/dashboards_nodejs_app/blob/master/routes/index.js

Let's not worry yet about persisting these or PUT vs POST or overwrite vs error. I want to get a simple demonstration of how a bundler can be written for Jupyter Notebook server to post a simple notebook to this API to get it to show up as a dashboard instantly.

Struggling to get make 'make run' and 'make run-tmpnb' to run on linux 14.04

Thanks for your efforts on this repo ... I'm watching it with interest ... however:

I ran make run ... which appeared to succeed in building images and then launching containers ... I'm not sure if the bower install step didn't get run properly or something ... the terminal logs would suggest maybe it's missing express (see logs lower down)

The screen that is meant to list notebooks was blank ... and this is what shows up for the simple notebook.

-- Starting proxy container
npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]~prestart: [email protected]
npm info lifecycle [email protected]~start: [email protected]

> [email protected] start /home/node/app
> node ./bin/www

GET / 302 49.128 ms - 64
GET /notebooks 200 240.792 ms - 281
STACK: Error: Not Found
    at /home/node/app/app.js:106:15
    at Layer.handle [as handle_request] (/home/node/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/node/app/node_modules/express/lib/router/index.js:312:13)
    at /home/node/app/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/home/node/app/node_modules/express/lib/router/index.js:330:12)
    at next (/home/node/app/node_modules/express/lib/router/index.js:271:10)
    at /home/node/app/node_modules/express/lib/router/index.js:618:15
    at next (/home/node/app/node_modules/express/lib/router/index.js:256:14)
    at Function.handle (/home/node/app/node_modules/express/lib/router/index.js:176:3)
    at router (/home/node/app/node_modules/express/lib/router/index.js:46:12)
GET /sw-import.js?baseURI=http%3A%2F%2Flocalhost%3A3000%2Fbower_components%2Fplatinum-sw%2Fplatinum-sw-register.html&clientsClaim=true&defaultCacheStrategy=networkFirst&importscript=http%3A%2F%2Flocalhost%3A3000%2Fbower_components%2Fplatinum-sw%2Fbootstrap%2Fsw-toolbox-setup.js&precache=&skipWaiting=true&version=1.0 404 13.974 ms - 878
STACK: Error: Not Found

I tried to stop and restart the dashboard-proxy container, so that I might troubleshoot things manually inside the container ... but ctrl-c of the make run process appears to kill and remove that container.

I don't have that much experience with make, but each time a run that command, it appears to build a whole new image ... which obviously isn't as slow as the first time, as it uses caches etc ... but is there scope for getting this working using docker-compose, especially once the images are built?

When running make run-tmpnb I also hit a hurdle on it complaining that port 8000 is already being used ... Although I have other docker containers running ... none of them is touching port 8000 ... does this 1005/node on port 8000 look like anything related to this set-up?

Step 21 : CMD start
 ---> Using cache
 ---> 0e8b480a3420
Successfully built 0e8b480a3420
4eb7912c7e6fd15e096e18a8036bd476fd1c27c2dc831c084037d6286a6d3662
Error response from daemon: Cannot start container 4eb7912c7e6fd15e096e18a8036bd476fd1c27c2dc831c084037d6286a6d3662: failed to create endpoint tmpnb-proxy on network bridge: Error starting userland proxy: listen tcp 0.0.0.0:8000: bind: address already in use
make: *** [run-tmpnb-proxy] Error 1
mccoole@mccoole-ubuntu-W520:~/Development/Tools/jupyter/dashboards_nodejs_app$ netstat -tulpn | grep 8000
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      1005/node

Update to stable ipywidgets/jupyter-js-widgets when available

Stop cloning from github in dockerfiles. Update package.json / bower.json to use the new version.

Simple demo sometimes says x is not defined

Like the message did not reach the kernel in order. Or at all.

%%javascript does not execute properly

Try https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_html_javascript.ipynb in the dashboard server. Works fine in the notebook server and the old Thebe setup. Here, the %%javascript cell does not render as a script tag but rather as a javascript object string.

Might be a bug in jupyter-js-output-area or transformime. Might be a bug in how we consume messages.

Initial implementation

The initial implementation should have the following properties

• fetch code from colocated notebook file on start
  • configurable path on disk
• use jupyter-js-services and jupyter-js-output-area, not thebe
• ability to filter websocket messages see issue #6
• supports one or more frontend browser client connections at a time and properly

Other issues will be opened about follow-on features.

Dashboard list shows hidden folders

Should we hide hidden files/folders in the dashboard list? Simple fix to give a bit of polish.

Apply rendered_html class to appropriately

In #83, we decided to quick fix the styling problems by slapping the class on all output areas. This will probably have side-effects. We should revisit and find a way to apply it only when proper, when the output is actual HTML. This probably involves looking at the mimetypes in the display_data and execute_result messages in the dashboards.js#messageHandlers. At present, that function only receives a model. Adding a class dynamically would require a reference to the view as well. (Not sure such a reference is a good thing thinking about MVC design.)

Error handling seems off

I'm now seeing a lot of these types of error in the server console:

STACK: Error: Not Found
    at /home/node/app/app.js:105:15
    at Layer.handle [as handle_request] (/home/node/app/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/home/node/app/node_modules/express/lib/router/index.js:312:13)
    at /home/node/app/node_modules/express/lib/router/index.js:280:7
    at Function.process_params (/home/node/app/node_modules/express/lib/router/index.js:330:12)
    at next (/home/node/app/node_modules/express/lib/router/index.js:271:10)
    at /home/node/app/node_modules/express/lib/router/index.js:618:15
    at next (/home/node/app/node_modules/express/lib/router/index.js:256:14)
    at Function.handle (/home/node/app/node_modules/express/lib/router/index.js:176:3)
    at router (/home/node/app/node_modules/express/lib/router/index.js:46:12)
GET /components/index.js.map 404 2.301 ms - 878

They don't really say much about what's going on. It just seems to be a generic error handler. @dalogsdon, I think we need to revisit how these error handlers are setup in the express app.

Style login/list pages

Currently we have default bootstrap style login and dashboard list. If we do issue #87, we could follow up with login and list styling to match. I put these in a separate issue as I feel they are less important than the navbar since they reside in their own pages which are not the main focus of dashboards.

Option to support https

Allow specification of the key and cert via nconf supported parameters.

Multiple linked widgets do not render, error in JS console

Bottom left panel in https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_layout_basic.ipynb should have 3 linked widgets. Only one of them seems to render. The other two generate "Could not create view" errors in the JS console

Error: Could not create view↵    at new Error (native)↵    at Error.WrappedError (http://127.0.0.1:3000/components/jupyter-js-widgets.js:31415:26)↵    at promiseRejection (http://127.0.0.1:3000/components/jupyter-js-widgets.js:31493:34

Do something less yucky at /

Right now, it's a vanilla page with text that says "List of notebooks".

• Pretty it up to have the banner like the dashboard pages have when auth is enabled.
• Show a bootstrap styled list of dashboard links
• If there's a dashboard notebook called index.ipynb, redirect immediately to that instead of showing the list?

Separate programmatic API from UI endpoint

• POST /notebooks -> move to POST /api/notebooks and keep as the programmatic way to add a new notebook
• GET / -> redirect and/or render /dashboards/index.ipynb if it exists or /dashboards if not.
• GET /dashboards -> UI list of all dashboards
• GET /dashboards/:name -> notebook rendered as a dashboard

Images do not resize to cell width

In the dashboard notebook plugin, images in cells will resize to cell width:

In the dashboard server, images are always full size:

This is using all Markdown cells with <img> elements.

Table styles missing

Try https://github.com/jupyter-incubator/dashboards/blob/master/etc/notebooks/test/test_widget_overlap.ipynb in the notebook dashboard preview and then here in the dashboard server. The table styling from the notebook environment is missing here.

Related? When the table is wider than the window width, it does not overflow or scroll. In the notebook server, it overflows outside the allocated grid cell. The behavior should be consistent with the notebook.

vs

Dashboard list shows sub directories.

The dashboard list page shows all contents of the data directory, which includes sub directories. However, selecting an entry in the list will attempt to load that as a notebook.

The GET /dashboards/* API does allow for sub directories, e.g. GET /dashboards/subdir/foo.ipynb.

Should we support sub directories in the list page or remove support from the GET API?

Max dashboard layout fixed width

User lays out dashboard in the (mostly) fixed-width environment of the notebook server. We should match that experience when we display the notebook here. Depending on the browser width, the dashboard container width should snap to some maximum. (See the notebook CSS or the dashboards_bundlers CSS for what those breakpoints are.)

Title of every dashboard page is "dashboard"

Gets confusing with lots of tabs. The filename would be fine. The first markdown header would be even better.

Remove use of bower, just use npm

We only use bower for Gridstack and its dependencies (jquery, jquery-ui, lodash). Should just simplify our build and install everything through npm.

Get a simple declarative widgets demo working

Try to get declarative widgets working on top of the bleeding edge ipywidgets we're using here already.

/cc @lbustelo

Logout link needs to appear on dashboard list page

Currently it's missing.

Option to support shared login

Support optional username and password configuration options via any nconf allowed method on server startup. If one is required, make sure the other is specified as well. If both are set, check whether or not all HTTP requests have a valid session cookie. If the user has authed (i.e., session cook is present and valid), service the HTTP request. If the user has not authed (i.e., the cookie does not exist, is expired, is invalid), show a basic login page prompting the user for the configured username and password. On successful login, set a temporary (1 day?) session cookie.

Use of client-side session cookies is fine here. (I think they're the express default still.)

Include a standalone demo

Add a notebook to this repo and instructions in the README about how to bring up an instance of the node app and have the notebook run as a dashboard. You'll probably want to use Docker because you'll need all of:

1. The notebook on disk
2. The node app
3. The kernel gateway
4. The kernel used in the notebook
5. Any other dependencies (matplotlib, node, npm, ...)

You could put all this in a single Docker container and launch two services for the demo (the node app + the kernel gateway). Or you could do a small docker-compose.yml with the kernel gateway running in one container (hopefully based on jupyter/docker-stacks#42 when it lands) and the node app in another.

The goal here is to show how the dashboard backend works independently of the bundler and the dashboard layout extension. Later, we can expand the README to link to those to paint the full picture.

---

TODO

• Create standalone demo notebook
• Create Docker container
• Replace jupyter-incubator/all-spark-kernels image with the minimal-kernel image.
• npm install jupyter-js-widgets instead of building it from source
• Use docker link command to link together the containers, instead of querying IP of kernel gateway container.

Allow dashboard clients to reconnect to running dashboards within a time window

Improve upon the immediate cleanup logic implemented in #25 to give the client time to try to reestablish its websocket connections.

Strawman: Set a timer in the nodejs app. If a client with the x-jupyter-session-id header returns before the timer fires, cancel the timer and continue forwarding messages to the associated kernel. If the timer goes off before a client reconnects, kill the kernel.

Of course, this also requires that the frontend JS attempt reconnections. It looks like jupyter-js-services already supports it, but we should check if it works in our use case.

Enable travis

See title.

Allow dashboard server to send auth headers to kernel gateway

1. tmpnb might be configured to require an auth token to request a KG server
2. KG server might be configured to require an auth token to request a kernel

Assume both of these keys can be set once at server start time via nconf, and just pass them along where appropriate. The keys should NOT be exposed to the frontend client.

• Pass auth token to kernel gateway
• Pass auth token to tmpnb
• Tests?

Only allow ipywidget comm channel messages to reach the kernel after initial top-to-bottom execute

Filter out any messages sent from the frontend client to the kernel at the backend server except for ipywidgets comm messages on the shell channel. Allowing any other message gives the user the ability to execute arbitrary code. (In fact, it's not yet know if locking down to comm messages only is good enough to prevent code execution or not yet, but it's a step in the right direction.)

See https://github.com/jupyter-incubator/dashboards/wiki/Deployed-Dashboard-Threat-Analysis

Support /api/spawn for tmpnb

The current implementation assumes a standalone kernel gateway or notebook server. This is equivalent to tmpnb: false mode in thebe. It also needs to support the /api/spawn mode of operation to launch containers on tmpnb (or compatible) clusters.

Reason: tmpnb with its configurable-http-proxy has the only robust mechanism for reaping kernels around. A single kernel gateway is fine for testing, but it leaks kernels.

• call /api/spawn on tmpnb to request a kernel gateway (make sure each client talks to its own kernel gateway instance)
• support KERNEL_GATEWAY_URL and KERNEL_CLUSTER_URL properties (see comments)
• support passing tmpnb auth token to /api/spawn

Use a real websocket library for decoding, filtering, encoding traffic

Right now we're using a somewhat home grown recipe that handles text frames well, but doesn't do anything with ping or pong or any of the other WS protocol op codes. This is biting us at the moment because the client's pongs are not making it to the kernel gateway in response to its pings, and so the KG kills the websocket connections after 90 seconds thinking they're dead.

I'm going to put in a quick hack fix for the pong problem, but we're pretty certain that it's not a robust solution. We need to move to something more robust eventually.

See sections 5.4 and 5.5.x of the websocket RFC https://tools.ietf.org/html/rfc6455#page-36

Indicate a problem when the kernel sends an error message

Strawman: some indicator in the nav bar at the top. Sort of like what notebook does when a kernel dies or a websocket disconnects or ... Don't hard code the indicator to kernel errors only though: it's very likely we're going to want to use that same indicator area to give other types of status too.