tmurph / jupyter-mode

Emacs tools to interact with Jupyter kernels.

jupyter.el

jupyter-mode is an Emacs minor mode for communicating with Jupyter kernels. You can start jupyter-console in a comint repl, create literate scripts with Org Babel, and obtain completion candidates from the kernel.

You should not start jupyter-mode from a mode hook, however. This will not set up the appropriate variables for a full session. Instead, either set up an Org Babel source block and call org-babel-initiate-session or start a session from a script buffer with jupyter-connect.

How To Install

These steps assume you understand the basics of Emacs’ load path and are comfortable compiling a simple C module with make.

Jupyter

Just use pip!

pip install jupyter

emacs-ffi

You’ll need to manually compile tromey’s emacs-ffi module.

git clone https://github.com/tromey/emacs-ffi.git
cd emacs-ffi
# tweak Makefile variables to suit your setup
make
cp ffi.el ffi-module.so /path/to/your/site/lisp

In the future I will automate this step with Cask.

jupyter-mode dependencies

jupyter.el requires dash and deferred for all operations. ob-juypter.el requires ob, and company-jupyter.el requires company.

If you don’t already have these packages, install them via your preferred method. I recommend M-x package-install.

jupyter-mode

Pull down the code.

git clone https://github.com/tmurph/jupyter-mode.git /path/to/your/site/lisp

Put the following lines in your init file and evaluate them.

(add-to-list 'load-path "/path/to/your/site/lisp/jupyter-mode")
(require 'jupyter)

If you’d like to enable Org Babel and Company support, add the following and evaluate them as well.

(require 'ob-jupyter)
(add-to-list 'org-src-lang-modes '("jupyter" . fundamental))

(require 'company-jupyter)
(add-to-list 'company-backends 'company-jupyter)

In the future I will include the whole package on MELPA.

Introductory Tutorials

Jupyter Mode

We’re going to write a script to numerically integrate a simple function.

For the purposes of this tutorial, we’re going to assume numpy and scipy are globally installed on your system. If you prefer to work in a virtual environment, scroll down for a guide to setting up a python kernel in a virtual environment.

Create a new file with M-x find-file RET my-script-name.py RET RET.

Visually inspect the major mode of the file; the file should be in Python mode.

If you are using the globally installed default kernel, say M-x jupyter-connect RET test-session-name RET RET. If you are using a python kernel you created in a virtual environment, say M-x jupyter-connect RET test-session-name RET your-virtualenv-kernel-name RET.

The file buffer should show that Jupyter minor mode is active, and Emacs should display a new buffer running the Jupyter console in Inferior Python mode.

Copy-and-paste the following code into the file buffer:

import numpy as np
from scipy.integrate import quad
from __future__ import print_function

def f(x):
    return (x-3)*(x-5)*(x-7)+85

a, b = 1, 8 # the left and right boundaries
N = 5 # the number of points
xint = np.linspace(a, b, N)
yint = f(xint)

integral, error = quad(f, a, b)
integral_trapezoid = sum( (xint[1:] - xint[:-1]) * (yint[1:] + yint[:-1]) ) / 2
print("The integral is:", integral, "+/-", error)
print("The trapezoid approximation with", len(xint), "points is:", integral_trapezoid)

Send the script code to the Jupyter console process with C-c C-c. You should see the following text appear after the console prompt:

The integral is: 565.2499999999999 +/- 6.275535646693696e-12
The trapezoid approximation with 5 points is: 559.890625

Org Babel Jupyter

We’re going to numerically integrate a simple function and plot a graph from an Org file via Org Babel.

We’re going to assume numpy, scipy, and matplotlib are globally installed on your system. If you prefer to work in a virtual environment, scroll down for a guide to setting up a python kernel in a virtual environment.

Ensure you have enabled Org Babel support for Jupyter by adding the following lines to your init file and evaluating them:

(require 'ob-jupyter)
(add-to-list 'org-src-lang-modes '("jupyter" . fundamental))

Create a new file with M-x find-file RET my-literate-script-name.org RET RET.

Visually inspect the major mode of the file; the file should be in Org mode.

If you are using a python kernel you created in a virtual environment, insert the following header at the top of the file:

#+PROPERTY: header-args:jupyter  :kernel your-virtualenv-kernel-name

Alternatively, if you are using the globally installed default kernel, don’t insert any such header.

Copy and paste the following text into the file buffer:

All source code blocks share a session, so this import will affect all later code.

#+BEGIN_SRC jupyter
  from __future__ import print_function
#+END_SRC

Define =f(x)=, the simple function we’re going to integrate.

#+BEGIN_SRC jupyter
  def f(x):
      return (x-3)*(x-5)*(x-7)+85
#+END_SRC

We’re also going to approximate the integral via the trapezoid rule.

#+BEGIN_SRC jupyter
  import numpy as np
  a, b = 1, 8 # the left and right boundaries
  N = 5 # the number of points
  xint = np.linspace(a, b, N)
  yint = f(xint)
#+END_SRC

This code block will print to stdout, which we capture with the =:results output= header argument.

#+BEGIN_SRC jupyter :results output
  from scipy.integrate import quad
  integral, error = quad(f, a, b)
  integral_trapezoid = sum( (xint[1:] - xint[:-1]) * (yint[1:] + yint[:-1]) ) / 2
  print("The integral is:", integral, "+/-", error)
  print("The trapezoid approximation with", len(xint), "points is:", integral_trapezoid)
#+END_SRC

This code block will produce a plot of the function and our trapezoid approximation to the integral.
We tell Org Babel to save the plot to a file (which Emacs can display inline) with the =:results file=
header argument.

#+BEGIN_SRC jupyter :results file
  %matplotlib inline
  import matplotlib.pyplot as plt

  x = np.linspace(0, 10, 200)
  y = f(x)

  plt.plot(x, y, lw=2)
  plt.axis([0, 9, 0, 140])
  plt.fill_between(xint, 0, yint, facecolor='gray', alpha=0.4)
  plt.text(0.5 * (a + b), 30,r"$\int_a^b f(x)dx$", horizontalalignment='center', fontsize=20);
#+END_SRC

Initiate the session by placing your cursor on any source code block and saying M-x org-babel-initiate-session RET.

Say M-x org-babel-execute-buffer RET to execute all code blocks in sequence and update the buffer with results. Alternatively, evaluate each code block manually by positioning your cursor anywhere on the block and pressing C-c C-c.

Company Jupyter

First, a word of caution. Completion may be too slow for on-the-fly use, as it requires several roundtrip requests of the kernel.

Currently, Jupyter completion only triggers on-the-fly after a dot, and results are cached. You may of course initiate completion at any time by saying M-x company-jupyter.

As with most company backends, press C-h on a completion candidate to temporarily pop up documentation in a separate buffer.

Working with Kernels

How to install ipykernel in a virtualenv

This is not about a part of jupyter.el, per se. Python virtual environments are an important part of many workflows, however, and I worry that the creation of Jupyter kernels in a virtual environment is still occasionally seen as black magic. This is how I do it.

Before executing the following lines in your shell, either set the environment variables $VENV_DIR and $KERNEL_NAME to your existing virtual environment directory and desired kernel name, or replace the references with your desired values.

cd "$VENV_DIR"
source bin/activate
pip3 install ipykernel
python3 -m ipykernel install --user --name "$KERNEL_NAME" --display-name "Python ($KERNEL_NAME)"

Now jupyter-connect will offer you the choice of KERNEL_NAME when you are starting a new session, and you may specify an Org Babel header argument of :kernel KERNEL_NAME to use that kernel for code block execution.

What about R and Julia?

Coming soon! Install any of the available Jupyter kernels on your system and ensure that you can see them at the terminal with jupyter kernelspec list.

You can reference those kernels from jupyter-connect or the Org Babel :kernel header argument. Code execution and Company completion should work just fine, however there is not yet much support for the inferior REPLs.

Reference Guides

Org Babel Header Arguments

Jupyter source blocks must include a :session header argument. A default value will be provided if you do not specify one.

You may specify a :kernel argument. The default is python.

If your code block will return a dataframe, specify :results dataframe in the header. This will trigger special output formatting based on the :colnames and :rownames arguments.

The first row of data (typically this is the dataframe column names) will be processed according to :colnames.

• if nil, don’t do any column name processing
• if “yes”, insert a line after the column names
• if “no”, exclude the column names
• default is “yes”

The first column of data (typically this is the dataframe index) will be processed according to :rownames

• if nil or “yes”, don’t do any index processing
• if “no”, exclude the index
• default is “no”

If your code block will produce a graph, specify :results file in the header. A random file name will be generated and the image will be put there. Alternatively, if the source block has a #+NAME then that will be used as the file name base. You may specify :output-dir to create the file in a specific directory. In instances where the kernel may return multiple image formats, you may specify :file-ext to select which one you want. Finally, you may specify the exact file name you want with :file.

Next Steps

• [X] actually connect roundtrip communication routines to Org Babel!
• [X] implement company completion with asynchronous completion requests
• [ ] write backend / frontend tests … maybe mock objects are my friend?
• [X] use kernel-info-request to determine the appropriate major mode for the inferior comint buffer
• [X] fix eldoc bug
• [ ] implement R and Julia support … the framework is there, just not the content
• [ ] instrument completion … how much can I get from speeding up my code?
• [X] fix comint startup bug
• [X] refactor / deep dive fixup PUB / SUB model
  • I’m getting bitten by the “slow subscriber” problem
  • http://zguide.zeromq.org/page:all#Getting-the-Message-Out
  • right now I’m just sleeping for a tenth of a second whenever I connect a new SUB socket … but that’s explicitly contra-indicated in the article
  • eventually, come back and implement their proposed solutions
  • some sort of proxy? where I have one proxy sub that lives forever, and a proxy pub that handles ephemeral connections?
  • no, it’s http://zguide.zeromq.org/page:all#Node-Coordination
  • wait, are you fucking insane? they explicitly build in a sleep(1) in there?!
  • yeah, okay, there is literally no way to do this without just sleeping for a bit
• [ ] maybe support fontification and eldoc in org source blocks?
  • eldoc of the code from the org file is maybe doable
    • [ ] remove the “user error” from naked jupyter-mode
    • [ ] set up a suitable eldoc-documentation-function
• [-] update documentation, see https://www.divio.com/en/blog/documentation/
  • [X] need a tutorial
    • learning-oriented
    • allows a newcomer to get started
    • is a lesson
  • [X] need a how-to guide
    • goal-oriented
    • shows how to solve a specific problem
    • is a series of steps
  • [ ] explanation
    • is understanding-oriented
    • explains
    • provides background and context
  • [ ] reference
    • information-oriented
    • describes the machinery
    • is accurate and complete
• [X] support remote kernels
  • a nifty tidbit … at the command line, an early -f file is overridden by a later --existing file
  • therefore, to get jupyter--acquire-session to do the right thing, just need to pass in the appropriate --existing file --ssh server
  • pretty easy to get these into org-babel-jupyter-initiate-session, just need to look for them in the babel params
  • in jupyter-connect we should probably just ask for them after asking for the kernelspec
  • fuck this is harder than I hoped. we gotta tweak all kinds of arguments, and when we finalize the session we gotta not delete the conn file.
  • like, this sounds like a job for factories and classes and such. but fuck. that. noise.
  • maybe just have a “ssh true” flag in the kernel object?
  • okay, so I think all my code looks right. for some weird reason, though, when I go to initialize the kernel it’s not creating the ssh connection file that I need to set up local ports? uh, what? like I can literally follow the code, pause right after it starts the inferior process, read the message at the top “use –existing blah-ssh.json” but I can’t find any blah-ssh.json file anywhere in my filesystem. wtf.
• [-] support org export
  • [X] support paragraphs -> markdown
    • [X] top-level paragraphs
      • this is basically done, but it’s exporting a final newline that I don’t particularly like. too tired to fix that now, though
    • [X] paragraphs in bulleted lists
  • [X] support code blocks -> code cell
  • [X] support #+CALL -> code cell
    • thankfully, these get their own specific parse element, babel-call. yay!
  • [X] support bold, italics, verbatim, and other light org markup
  • [X] support bulleted lists
  • [ ] support numbered lists
  • [-] support links
    • [X] http, https, mailto
    • [X] relative file links
    • [ ] internal fuzzy links
      • per this blogpost I can definitely make this work
      • but I think I’ll need to do some parse tree surgery in advance. looks like, to do it right, I wanna add a special paragraph section before the target that just contains an HTML anchor tag, or something like that. or maybe before we even parse the buffer I just insert the ATTR_HTML or whatever?
      • ugh, there’s a problem where the space before a link gets eaten in a plain paragraph. cool.
  • [-] export code results
    • [-] export images
      • not really done, but so much progress. there’s a lot of hard-coding going on (only PNG file endings, always assumes 640x480 image size) but that stuff should be easier to modify as we go along. the real hard part was pulling the image data through at all.
    • [X] export output streams / verbatim results
      • jfc this was hard, had to munge the parse tree. interesting tool, tho
    • [ ] export dataframes
  • [-] export top-level structure
    • [X] list-of-cells
    • [X] version info
    • [-] metadata
      • this is sorta there. we can export arbitrary strings with a keyword (yay!) but I’m not satisfied with the implementation yet. how to export stuff like that git filter setting? should we bother with kernel info?
  • [X] run multiple paragraphs / paragraph-like elements together
    • Org parser treats paragraphs, bulleted lists, source blocks, etc all at the same level
    • so we naively just create separate notebook cells for each
    • what if we wanted a string of paragraphs and lists to merge into one cell?
    • could maybe do some parsing of the tree behind the scenes
    • need a delimiter … org parser has the notion of a “horizontal rule”, it’s just 5 or more dashes
    • let’s just use the line break \\ feature
    • one line break means “merge the next paragraph / plain list”
    • extra line breaks include additional spaces