Project Jupyter not only revolutionizes data-heavy research across domains - it also boosts personal productivity for problems on a much smaller scale. Due to openness it is an amazing platform for exploring concepts and learning new things.
I started setting up a Jupyter Notebook Server on a Raspberry Pi following this blog post by Arun Durvasula. Convinced of the potential of the platform I started to follow the development.
What started as a personal exercise soon turned taught me a great deal about the underlying architcture. Given Jupyter's complexity, speed of growth and scale it is remarkable that such a system runs fine on a Raspberry Pi.
This repository isn't really anything genuine: I owe big thanks to many contributors in the Jupyter, Raspberry Pi, Linux, Python, Julia and greater Open Source communities for providing all the beautiful building blocks - small and large.
- Rather than installing the latest version of Python as I did the past, I decided that the new version would use the latest Python 3 version supported in Raspbian - as of this writing Python 3.5.3.
- Whilst this seems to be a step backwards, it is a in fact a giant step forward as we benefit from significant installation speedups made possible by the recently released piwheels project.
- The scripts work across the entire range of Raspberry Pis - perhaps with the exception of the early models with just 256MB of memory.
- Python support for GPIO, Sense HAT and PICAMERA is installed without the earlier worries of breaking things on system level.
- All Python modules are pip installed into a virtual environment following advice found online: You should never use
sudo pip install
-NEVER. Well I did this in the past and it certainly had me and users confused. We have to learn certain things the hard way to really appreciate the benefits of doing them right. It is worth reading up on this in this blogpost. - We now install Python packages into a virtual environment created with
venv
using arequirements.txt
file. This really achieves a more maintainable setup, opens up more possibilities and hopefully makes this project more useful for the Raspberry Pi and Jupyter communities. - Python 3, Julia and Bash kernels are installed and configured during installation.
- JupyterLab is installed and ready to use.
- a Raspberry Pi model of your choice complete with micro-usb power-supply - I recommend a Raspberry Pi 3 but the setup should work across the entire range of Pi models, perhaps with the exception of the very early models that featured only 256MB of memory. I tested on a ZeroW, 1, 2 and 3.
- a (micro) SD card with 16GB capacity or more to suit your Pi model with Raspbian Stretch Lite installed and configured to permit access via ssh as user pi.
- an ethernet or wifi connection for the Pi
- internet access on the Pi
- a computer to carry out the installation connected to the same network as the Pi
- LESS TIME THAN EVER BEFORE due to the recent release of piwheels. Users new to this project might argue that the setup is still time-consuming. Believe me: In the past 6 hours+ were not uncommon and installing the system on a Raspberry Pi 1 was not impossible but required quite some patience and time.
-
ssh into your Raspberry Pi with the the fresh install of Raspbian Stretch Lite as user pi. Then run
sudo raspi-config
and set the memory split to 16MB, expand the file-system and set a new password for the user pi. When done, reboot and log in again via ssh. -
Next install
git
:
sudo apt install -y git
- With preparations out of the way clone this repository into the home directory of user pi
git clone https://github.com/kleinee/jns
- change into the new directory
jns
just created withgit
:
cd ~/jns
- To increase size of swap_file to 2048MB open
/etc/dphys-swapfile
and changeCONF_SWAPSIZE=100
toCONF_SWAPSIZE=2048
sudo vi /etc/
Technically you now just run sudo apt ./inst_jns.sh
which is the installer script that combines the steps described below. If you follow along I assume that you run all scripts from inside this directory.
sudo ./prep.sh
A couple of packages from the Raspbian repository are required during installation and later for a some Python packages tp work properly. The script just fetches these packages and installs them.
#!/bin/bash
# script name: prep.sh
# last modified: 2018//01/07
# sudo: yes
script_name=$(basename -- "$0")
if ! [ $(id -u) = 0 ]; then
echo "usage: sudo ./$script_name"
exit 1
fi
apt update
apt -y upgrade
apt -y install pandoc
apt -y install libxml2-dev libxslt-dev
apt -y install libblas-dev liblapack-dev
apt -y install libatlas-base-dev gfortran
apt -y install libtiff5-dev libjpeg62-turbo-dev
apt -y install zlib1g-dev libfreetype6-dev liblcms2-dev
apt -y install libwebp-dev tcl8.5-dev tk8.5-dev
apt -y install libharfbuzz-dev libfribidi-dev
apt -y install libhdf5-dev
apt -y install libnetcdf-dev
apt -y install python3-pip
apt -y install python3-venv
apt -y install libzmq3-dev
./inst_stack.sh
- This creates a virtual Python 3 environment '/home/pi/.venv/jns' and activates it temporarily
- It then updates
pip3
to the latest version available version from the Python package repository before it processes therequirements.txt
file line by line. - This is a workaround to prevent
pip
from failing if one or more requirements listed fail to install.
#!/bin/bash
# script name: inst_stack.sh
# last modified: 2018/01/14
# sudo: no
script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"
if [ $(id -u) = 0 ]
then
echo "usage: ./$script_name"
exit 1
fi
# create virtual environment
if [ ! -d "$venv" ]; then
python3 -m venv $env
fi
# activate virtual environment
source $env/bin/activate
pip3 install pip==9.0.0
pip3 install -U pip
pip3 install -U setuptools
cat requirements.txt | xargs -n 1 pip3 install
./conf_jupyter.sh
The scripts generate a jupyter notebook configuration directory and in it a file called jupyter_notebook_config.py
that holds the configuration settings for our notebook / lab server. We also create a folder notebooks in the home directory of user pi
as the notebook_dir
for our server. In the configuration file, we apply the following changes:
- tell jupyter not to sart a browser upon start - we access the server from a remote machine on the same network
- set the IP address to '*'
- set the port for the notebook server to listen to 8888
- enable
mathjax
for rendering math in notebooks - set the notebook_dir to
~/notebooks
- use the password hash for the default server password
jns
NOTE: This setup still uses password authentication. If you prefer token-based authentication as introduced with notebook you will have to change settings in the config file /home/pi/.jupyter/jupyter_notebook_config.py
to. Documentation of possible configuration settings can be found [here](http://minrk notebook.readthedocs.io/en/latest/notebook.html).
After the basic configuration the script activates the bash kernel and activates extensions for Jupyter Notebook and Jupyter Lab. At the JupyterLab end this requires intstallation of nodejs
followed by installation of the underlying JS infrastructure which is a bit time-consuming but ultimately allows us to use ipywidgets
and bqplot
.
#!/bin/bash
# script name: conf_jupyter.sh
# last modified: 2018/02/22
# sudo: no
script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"
if [ $(id -u) = 0 ]
then
echo "usage: ./$script_name"
exit 1
fi
# activate virtual environment
source $env/bin/activate
# generate config and create notebook directory
# if notebook directory exists, we keep it (-p)
# if configuration file exeists, we overwrite it (-y)
jupyter notebook -y --generate-config
cd $home
mkdir -p notebooks
target=~/.jupyter/jupyter_notebook_config.py
# set up dictionary of changes for jupyter_config.py
declare -A arr
app='c.NotebookApp'
arr+=(["$app.open_browser"]="$app.open_browser = False")
arr+=(["$app.ip"]="$app.ip ='*'")
arr+=(["$app.port"]="$app.port = 8888")
arr+=(["$app.enable_mathjax"]="$app.enable_mathjax = True")
arr+=(["$app.notebook_dir"]="$app.notebook_dir = '/home/$(logname)/notebooks'")
arr+=(["$app.password"]="$app.password = 'sha1:5815fb7ca805:f09ed218dfcc908acb3e29c3b697079fea37486a'")
# apply changes to jupyter_notebook_config.py
for key in ${!arr[@]};do
if grep -qF $key ${target}; then
# key found -> replace line
sed -i "/${key}/c ${arr[${key}]}" $target
else
# key not found -> append line
echo "${arr[${key}]}" >> $target
fi
done
# install bash kernel
python3 -m bash_kernel.install
# install extensions
jupyter serverextension enable --py jupyterlab
jupyter nbextension enable --py widgetsnbextension --sys-prefix
jupyter nbextension enable --py --sys-prefix bqplot
# activate clusters tab in notebook interface
/home/pi/.venv/jns/bin/ipcluster nbextension enable --user
# install nodejs and node version manager n
curl -L https://git.io/n-install | bash -s -- -y
. /home/pi/.bashrc
jupyter lab clean
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install bqplot
jupyter labextension install jupyterlab_bokeh
Since we use a virtual environment to install Python modules, we need to activate this environment before we can start our server:
source /home/pi/.venv/jns/bin/activate
The prompt will change to indicate successfull activation preceding pi@hostname:
with the envireonment name - in our case (jns)
. With hostname set to zerow
it looks like this:
(jns) pi@zerow:~ $
To start the server just type jupyter notebook
or jupyter lab
To access the server form a webbrowser on a computer running on the same network as your Raspberry Pi just open a browser and use the Pi's IP address / port 8888 as the url.
xxx.xxx.xxx.xxx:8888
Change `xxx.xxx.xxx.xxx' to the IP address of the Raspberry Pi.
During the configuration the default password for the server was set to jns
. This can be changed by:
(jns) pi@zerow:~ $ jupyter notebook password
Enter password: ****
Verify password: ****
sudo ./inst_tex.sh
- TeX (and Pandoc) are used under the hood to convert Jupyter notebooks to other formats including PDF.
- Whilst not strictly necessary if no PDF export is rquired, I still recommend to run this step.
#!/bin/bash
# script name: inst_tex.sh
# last modified: 2018/01/26
# sudo: yes
script_name=$(basename -- "$0")
if ! [ $(id -u) = 0 ]; then
echo "usage: sudo ./$script_name"
exit 1
fi
#------------------------------------------------------
apt install -y texlive
apt install -y texlive-latex-extra
apt install -y dvipng
apt install -y texlive-xetex
apt install -y latexmk
#------------------------------------------------------
sudo ./inst_julia.sh
-
Julia is a relatively new high-level, high-performance dynamic programming language for numerical computing trying to combine the ease of Python with the speed of C. Thanks to the efforts of the Raspberry Pi community
Julia 0.6.0
is availabel in the Raspbian Stretch Repository. It is really worth a try as the language is a rising star in scientific computing biting into the userbase of Matlab. -
IJulia is the kernel required for Jupyter Notebook / Lab. Backgroud information on Julia on the Raspberry Pi can be found here.
-
JuliaBerry is a GitHub organization maintaining various repositories with Julia packages for the Raspberry Pi. Sense HAT and GPIO support are definitely worth to explore.
#!/bin/bash
# script name: inst_julia.sh
# last modified: 2018/01/14
# sudo: yes
env=/home/pi/.venv/jns
script_name=$(basename -- "$0")
if ! [ $(id -u) = 0 ]; then
echo "usage: sudo ./$script_name"
exit 1
fi
env=/home/pi/.venv/jns
apt -y install julia
su pi <<EOF
source $env/bin/activate
julia -e 'Pkg.add("IJulia")'
julia -e 'using IJulia'
EOF
setting up Python s support for GPIO pins, PICAMERA module and Sense HAT hardware in our virtual environment is almost as simple as one would commonly do without such environment.
#!/bin/bash
# script name: inst_pi_hardware.sh
# last modified: 2018/01/14
# sudo: no
script_name=$(basename -- "$0")
env="/home/pi/.venv/jns"
if [ $(id -u) = 0 ]
then
echo "usage: ./$script_name"
exit 1
fi
# activate virtual environment
source $env/bin/activate
git clone https://github.com/RPi-Distro/RTIMULib
cd ./RTIMULib/Linux/python/
python3 setup.py build
python3 setup.py install
cd /home/pi/jns
rm -rf RTIMULib
pip3 install sense-hat
pip3 install picamera
pip3 install gpiozero
This script is just convenience - it executes the individual steps described above in the order necessary.
#!/bin/bash
# script name: inst_jns.sh
# last modified: 2018/01/14
# sudo: yes
script_name=$(basename -- "$0")
if ! [ $(id -u) = 0 ]; then
echo "usage: sudo ./$script_name"
exit 1
fi
./prep.sh
./inst_tex.sh
./inst_pi_hardware.sh
sudo -u pi ./inst_stack.sh
sudo -u pi ./conf_jupyter.sh
./inst_julia.sh
- just run
sudo apt update && sudo apt -y upgrade
-
activate the virtual environment with
source /home/pi/.venv/jns/bin/activate
-
list outdated packages with
pip3 list --outdated --format='legacy'
-
Update
package
withpip3 install -U package
wherepackage
is the package you want to update.