jupyter / colaboratory

[deprecated] Jupyter CoLaboratory, goto google colab now

Jupyter

Read this in other languages: English, Español, Português, Français

Jupyter metapackage for installation and documents

Documentation structure

This documentation uses the Sphinx documentation engine.

The documentation is located in the docs/source folder. When you build the documentation, it will be placed in the docs/build folder. It is written in a combination of reStructuredText and MyST Markdown.

Build the documentation locally

There are a few ways to build the documentation; see below for instructions:

Build the documentation automatically with nox

The easiest way to build the documentation locally is by using the nox command line tool. This tool makes it easy to automate commands in a repository, and we have included a docs command to quickly install the dependencies and build the documentation.

To build and preview the site locally, follow these steps:

1. Clone this repository.

   $ git clone https://github.com/jupyter/jupyter
   $ cd jupyter

2. Install nox

   $ pip install nox

3. Run the docs command

   $ nox -s docs

This will install the needed dependencies in a virtual environment using pip. It will then place the documentation in the docs/build/html folder. You may explore these HTML files in order to preview the site.

Create a live server to automatically preview changes

There is another nox command that will do the above, and also create a live server that watches your source files for changes, and auto-builds the website any time a change is made.

To start this live server, use the following nox command:

$ nox -s docs-live

When the build is finished, go to the URL that is displayed. It should show a live preview of your documentation.

To stop serving the website, press Ctrl-C in your terminal

Build the documentation manually

To build the documentation manually, follow these steps:

First, install the miniconda Python distribution.

Next, navigate to the /docs directory and create a conda environment:

conda env create -f environment.yml

Activate the environment:

source activate jupyter_docs

Build the docs using Sphinx with the following commands:

make clean
make html

The docs will be built in build/html. They can be viewed by opening build/html/index.html or starting an HTTP server and navigating to 0.0.0.0:8000 in your web browser.

python3 -m http.server

Releasing the jupyter metapackage

Anyone with push access to this repo can make a release of the Jupyter metapackage (this happens very rarely). We use tbump to publish releases.

tbump updates version numbers and publishes the git tag of the version. Our GitHub Actions then build the releases and publish them to PyPI.

The steps involved:

1. Install tbump: pip install tbump
2. Tag and publish the release tbump $NEW_VERSION.

That's it!

colaboratory's Issues

"Share" does not work in Chrome App

If a user un-installs Chrome, the cached auth token still exists

After a user un-installs Chrome from Drive, the App will still think it has an oauth token, because the token is cached. So Chrome App will not prompt the user to re-install.

This is a minor bug since we don't expect people to uninstall.

Rename branches in repo

v2_IPython20 -> master
master -> v1

https requests do not work in PNaCl kernel

HTTPS requests result in an error inside the PNaCl kernel. Example:

Input:

urllib2.urlopen('https://www.google.com')

Error (not full stack trace):

/usr/lib/python2.7/urllib2.py in https_open(self, req)
   1220 
   1221         def https_open(self, req):
-> 1222             return self.do_open(httplib.HTTPSConnection, req)
   1223 
   1224         https_request = AbstractHTTPHandler.do_request_

/usr/lib/python2.7/urllib2.py in do_open(self, http_class, req)
   1182         except socket.error, err: # XXX what error?
   1183             h.close()
-> 1184             raise URLError(err)
   1185         else:
   1186             try:

URLError: <urlopen error [Errno -1] Invalid value for `ai_flags' field.>

@matthewturk, I'm assigning this to you as you know more about this than me, but I'm happy to help debug this issue.

Create Cloud Console project and Chrome Webstore console.

This is so the Chrome App is publicly installable, and owned by the Jupyter project.

Test Chrome App after all recent changes

Do a fresh install of the whole Chrome App and verify functionality. Add new issues as specific problems are identified.

Purge repo of large binary blobs

Over the history of the repo, some binary blobs from PNaCl builds may have made their way into the git history. We should run git filter-branch and remove them to make the repo as small as possible before making it public.

Create icons, screenshots, and promotional images for Chrome App

Capture mouse clicks (selection) and shift-enter (run) for iframes

Also @KesterTong

Delete Cloud Console Project containing old Client IDs

Previously we were using client IDs in a different project from Google's internal build. However this caused realtime API conflicts, so all Client IDs are now moved into the same project as the one used by Google's internal build.
Deleting the project containing the old Client IDs will cause any forks using the old Client IDs to stop working, but will prevent any further realtime conflicts

Make "Connect to Kernel" process easier

Create a new entrypoint, colaboratory.py, along the lines of SingleCellServer.py

Currently, the Colab notebook static code piggybacks off the IPython tornado web server. A better way is to completely rewrite the frontend handlers along the lines of SingleCellServer.py.

This would allow for running IPython without a shell script run.sh, and also decouple the server from the IPython profile.

Basic bokeh support via output_notebook()

When trying to execute this in a cell:
from bokeh.plotting import *
output_notebook()

I get this:

RuntimeError Traceback (most recent call last)
in ()
1 from bokeh.plotting import *
----> 2 output_notebook()

/home/dave/anaconda/lib/python2.7/site-packages/bokeh/plotting.pyc in output_notebook(url, docname, session, name)
155 else:
156 from . import load_notebook
--> 157 load_notebook()
158 global _default_notebook
159 _default_notebook = True

/home/dave/anaconda/lib/python2.7/site-packages/bokeh/init.pyc in load_notebook(resources, verbose, force, skip)
36 pass
37 if not notebook:
---> 38 raise RuntimeError('load_notebook() only works inside an IPython Notebook.')
39
40 import IPython.core.displaypub as displaypub

RuntimeError: load_notebook() only works inside an IPython Notebook.

The entry point for colaboratory should be /welcome and /notebook

We need to remove the static and html prefix for user visible urls rather than /static/welcome.html and /static/notebook.html

Fix connection issues at start

• default address is https
• pops up unnecessary authentication window.

Open Source push - Agenda (7/27 and 7/30)

put agenda here:

• setup colaboratory.jupyter.org to use as an entry point for the webapp. Validate that someone other than @fperez can do it with the common dreamhost credentials. See #20.
• review history of the current repo and see whether we need to filter big binaries out of it first. See #26.
• test install instructions/readme on a clean machine for accuracy. See #46.
• list any current deviations of file format and sketch out future plans for each.
• include in README a clear explanation of the differences in experience/capabilities between this and a 'regular' ipython install.
• upgrade coLaboratory to v2
• create plan for adding collaborative functionality within core IPython
• Remove modified IPython source code from repo and figure out a better way of doing that

Confusion in the execution model while working collaboratively

Right now, when two or more users are editing a single notebook collaboratively, the result can be quite confusing when execution is also mixed in: everyone sees a single, common output view, but that view is really the mix of the states of everybody's kernels.

The end-user experience can be quite confusing.

We'll need to sort out a simpler execution model, perhaps with an 'execution baton' that can be passed from one user to another but can only be held by one person at a time.

Change Client IDs

Current Client ID's belong to a different Google Cloud Console project to the internal Google version. This will cause realtime conflicts.

Testing framework

Figure out how to test this code in a way that integrates with the rest of Jupyter testing machinery.

Google Closure library submodule does not work

The latest merge changes the location of the submodule containing the Google Closure library. However cloning the repo now fails to create the Closure library submodule, even when using the --recursive flag.

I believe this can be fixed by deleting the .gitmodules file and then rerunning the git submodule add command, but I'm not sure if there were reasons for not doing this.

Add colabtools library to Chrome App

See #34.

Google Drive OAuth: Origin Mismatch

Got this error when I tried to install coLab in Google Drive. I know I need to change the OAuth tokens Google provides - just can't figure out where I change those, in the code.

Add to build process pulling in NaCl ports binaries.

This is so we can build the Chrome App without stuffing large binary blobs inside the git source repository.

This task subsumes #3.

Use IPython-version-independent code for tab-completion

Right now, the tab completion only works with IPython 1.x because it doesn't use the version-independent code.

getting started link fails due to permissions

The "getting started" link on colaboratory.jupyter.org links to https://sites.google.com/a/google.com/colaboratory/home/running-colab-and-kernels

which fails with "You need permission"

Cannot read property 'modal' of undefined on starting local colaboratory

I installed the coLaboratory Drive app from http://colaboratory.jupyter.org/welcome/.

I then ran:

git clone --recursive https://github.com/jupyter/colaboratory
cd colaboratory
pip install -r requirements.txt
python -m colaboratory

Visiting http://127.0.0.1:8888/welcome, I click "Create" to create a new notebook.

I see a popup with error Cannot read property 'modal' of undefined on starting local colaboratory:

but then the kernel seems to work fine:

Create script to create debug/release versions of Chrome App

Chages:

• App ID in filepicker.js
• Client ID in manifest.json

chrome extension: C++ Error >Aborting: _Unwind_RaiseException called

Installed as instructed, local IPython technique worked fine. NACL failed. Running OSX 10.9.2

nacl>Access: /home == -1
 pnacl_kernel.js:92
nacl>Successfully created /home
 pnacl_kernel.js:92
nacl>Access: /home/zeropy == -1
 pnacl_kernel.js:92
nacl>Successfully created /home/zeropy
 pnacl_kernel.js:92
nacl>Mounted html5fs, moving on.
 pnacl_kernel.js:92
nacl>Access: /usr == -1
 pnacl_kernel.js:92
nacl>Successfully created /usr
 pnacl_kernel.js:92
nacl>Access: /tmp == -1
 pnacl_kernel.js:92
nacl>Successfully created /tmp
 pnacl_kernel.js:92
nacl>Access: /etc == -1
 pnacl_kernel.js:92
nacl>Successfully created /etc
 pnacl_kernel.js:92
nacl>Access: /mnt == -1
 pnacl_kernel.js:92
nacl>Successfully created /mnt
 pnacl_kernel.js:92
nacl>Access: /mnt/http == -1
 pnacl_kernel.js:92
nacl>Successfully created /mnt/http
 pnacl_kernel.js:92
nacl>Grabbing env var.
 pnacl_kernel.js:92
nacl>Got something.
 pnacl_kernel.js:92
nacl>DATA URL: zeropy_20140520.tar.gz
 pnacl_kernel.js:92
nacl>Extracting ZeroPy environment.
 pnacl_kernel.js:92
nacl>Extracting /mnt/http/zeropy_20140520.tar.gz
 pnacl_kernel.js:92
nacl>Asked for O_RDONLY
 pnacl_kernel.js:92
nacl>RETURNING GZF: 1055416664
 pnacl_kernel.js:92
nacl>Opened '/mnt/http/zeropy_20140520.tar.gz' successfully (into /usr)
 pnacl_kernel.js:92
nacl>Aborting: _Unwind_RaiseException called (C++ exception handling is disabled)
 pnacl_kernel.js:92
nacl>ted:1 

Add CORS headers to Colab webapp

CORS headers are required to connect from colaboratory.jupyter.org to a local kernel.

Check in PNaCl kernel built in naclports

Check in the files for the PNaCl kernel into chrome/pnacl directory. This will ensure the Chrome App can be built from the repo, and is using the latest PNaCl build from naclports.

Document clearly current execution/collab model

In the longer term, see #22. For now, just document how it currently works.

colabtools.drive should be able to load files created elsewhere

Make welcome page in Chrome App same as for web app

Directory Sturctures should work for linking notebooks

Allowing cross-notebook links when stored in GDrive to work.

Also @kayur.

Directory structures should work for loading data

Provide user apis for loading data from GDrive.

Also @kayur.

Kernel cannot reconnect after failed connection

If the user tries to connect to the wrong kernel address, then attempts to connect to a correct address, the second connection will still fail, as the function Session.delete() fails at session.js line 79.

This problem would be easily solved by changing that line, but since it is IPython code, we should instead try to understand why this is happening and address the root cause.

FilePicker API does not work in Chrome App

The FilePicker API is used to display an "open" dialog box, to open files from Google Drive. This is used in both the welcome screen and the regular notebook window (file > open). Currently the FilePicker API does not work in the Chrome App, for reasons unknown.

Some drive functionality doesn't work when mixing http and https

When running on the local server and trying to share, I get this error:

 
Refused to display 'https://drive.google.com/share?id=0B0T... in a frame 
because it set 'X-Frame-Options' to 'SAMEORIGIN'.

remove Google-y references

welcome.html

• link to this internal Google page in static/frontend/welcome.html
• "Learn More" is a go/colab link
• library logos refer to dremel and big table in static/v2/img/library-logos.png
• Feedback menu -> Report bug should point to the issues for the new location of this repo

Reconcile kernel connection with IPython 2.0 protocol

Right now the same notebook will connect to a new kernel each time a connection is made, rather than connecting to a single kernel for that notebook, and only launching a new kernel if no kernel for that notebook is running. This is because we don't use session.js, only kernel.js, in the front end.

Host the static content at colaboratory.jupyter.org

Create a site with all the same static content that the chrome app needs, so that Drive-hosted notebooks can be rendered without users having to install anything other than registering colaboratory.jupyter.org as a known Drive app.

Test install instructions/readme on a clean machine for accuracy

Cell-level comments & Notify on unread comment.

Add commenting support to notebooks hosted on GDrive (like other docs apps), with notifications.

Establishing procedures for pull requests?

Now would be a good time to establish conventions for changes to the repo.

I would propose the following procedure: every PR should be approved by

• an Jupyter developer, to ensure changes remain compatible with Jupyter code and vision,
• one of @KesterTong or @matthewturk, to prevent breakages of the Chrome App, and
• one of @kayur or @marksandler2, to prevent breakages of the Classic web app.

If this is excessive, we can just have anyone from the above approve PR's, on the condition they get approval from one of the above people where needed.

How does that sound? @fperez @ellisonbg @minrk @matthewturk @kayur @marksandler2

Move "colabtools" library to git repo

Colabtools is a set of Python libraries that provide extra functionality within Colaboratory, such as plotting with d3.

Chrome App needs to disallow install for chrome versions below 35.1975

Chrome App is broken badly for early releases, we need to prevent it from running on those old versions

Compile JS using closure compiler

In the build process, instead of copying the sources, run the closure compiler first before serving individual JS files.

Does the README need a stage for running bundle_static

It wasn't clear to me whether running bundle_static from the command line is now part of the install process.

Support loading local files from Chrome App

Detect when connected to localhost and use this port when connecting to kernel

When determining default kernelUrl, if the window.location is localhost or 127.0.0.1, use the port of the window.location instead of :8888 as the port.