ipython-books / cookbook-2nd

IPython Cookbook, Second Edition, by Cyrille Rossant, Packt Publishing 2018

IPython Cookbook, Second Edition (2018)

IPython Interactive Computing and Visualization Cookbook, Second Edition (2018), by Cyrille Rossant, contains over 100 hands-on recipes on high-performance numerical computing and data science in the Jupyter Notebook.

This repository contains the sources of the book (in Markdown, CC-BY-NC-ND license).

▶ Get the code as Jupyter notebooks
▶ Get the Google Chrome extension to see LaTeX equations on GitHub
▶ Buy the book

Contents

Chapter 1 : A Tour of Interactive Computing with Jupyter and IPython

• 1.1. Introducing IPython and the Jupyter Notebook
• 1.2. Getting started with exploratory data analysis in the Jupyter Notebook
• 1.3. Introducing the multidimensional array in NumPy for fast array computations
• 1.4. Creating an IPython extension with custom magic commands
• 1.5. Mastering IPython's configuration system
• 1.6. Creating a simple kernel for Jupyter

Chapter 2 : Best practices in Interactive Computing

• 2.1. Learning the basics of the Unix shell
• 2.2. Using the latest features of Python 3
• 2.3. Learning the basics of the distributed version control system Git
• 2.4. A typical workflow with Git branching
• 2.5. Efficient interactive computing workflows with IPython
• 2.6. Ten tips for conducting reproducible interactive computing experiments
• 2.7. Writing high-quality Python code
• 2.8. Writing unit tests with py.test
• 2.9. Debugging code with IPython *

Chapter 3 : Mastering the Jupyter Notebook

• 3.1. Teaching programming in the Notebook with IPython blocks
• 3.2. Converting a Jupyter notebook to other formats with nbconvert
• 3.3. Mastering widgets in the Jupyter Notebook
• 3.4. Creating custom Jupyter Notebook widgets in Python, HTML, and JavaScript
• 3.5. Configuring the Jupyter Notebook *
• 3.6. Introducing JupyterLab

Chapter 4 : Profiling and Optimization

• 4.1. Evaluating the time taken by a command in IPython *
• 4.2. Profiling your code easily with cProfile and IPython
• 4.3. Profiling your code line-by-line with line_profiler
• 4.4. Profiling the memory usage of your code with memory_profiler
• 4.5. Understanding the internals of NumPy to avoid unnecessary array copying
• 4.6. Using stride tricks with NumPy
• 4.7. Implementing an efficient rolling average algorithm with stride tricks
• 4.8. Processing large NumPy arrays with memory mapping
• 4.9. Manipulating large arrays with HDF5 *

Chapter 5 : High-Performance Computing

• 5.1. Knowing Python to write faster code
• 5.2. Accelerating pure Python code with Numba and just-in-time compilation
• 5.3. Accelerating array computations with Numexpr
• 5.4. Wrapping a C library in Python with ctypes
• 5.5. Accelerating Python code with Cython
• 5.6. Optimizing Cython code by writing less Python and more C
• 5.7. Releasing the GIL to take advantage of multi-core processors with Cython and OpenMP
• 5.8. Writing massively parallel code for NVIDIA graphics cards (GPUs) with CUDA
• 5.9. Distributing Python code across multiple cores with IPython
• 5.10. Interacting with asynchronous parallel tasks in IPython
• 5.11. Performing out-of-core computations on large arrays with Dask
• 5.12. Trying the Julia programming language in the Jupyter Notebook *

Chapter 6 : Data Visualization

• 6.1. Using matplotlib styles
• 6.2. Creating statistical plots easily with seaborn
• 6.3. Creating interactive Web visualizations with Bokeh and HoloViews
• 6.4. Visualizing a NetworkX graph in the Notebook with D3.js
• 6.5. Discovering interactive visualization libraries in the Notebook *
• 6.6. Creating plots with Altair and the Vega-Lite specification

Chapter 7 : Statistical Data Analysis

• 7.1. Exploring a dataset with pandas and matplotlib
• 7.2. Getting started with statistical hypothesis testing — a simple z-test
• 7.3. Getting started with Bayesian methods
• 7.4. Estimating the correlation between two variables with a contingency table and a chi-squared test
• 7.5. Fitting a probability distribution to data with the maximum likelihood method
• 7.6. Estimating a probability distribution nonparametrically with a kernel density estimation
• 7.7. Fitting a Bayesian model by sampling from a posterior distribution with a Markov Chain Monte Carlo method
• 7.8. Analyzing data with the R programming language in the Jupyter Notebook *

Chapter 8 : Machine Learning

• 8.1. Getting started with scikit-learn
• 8.2. Predicting who will survive on the Titanic with logistic regression *
• 8.3. Learning to recognize handwritten digits with a K-nearest neighbors classifier
• 8.4. Learning from text — Naive Bayes for Natural Language Processing
• 8.5. Using support vector machines for classification tasks
• 8.6. Using a random forest to select important features for regression
• 8.7. Reducing the dimensionality of a dataset with a principal component analysis *
• 8.8. Detecting hidden structures in a dataset with clustering

Chapter 9 : Numerical Optimization

• 9.1. Finding the root of a mathematical function *
• 9.2. Minimizing a mathematical function
• 9.3. Fitting a function to data with nonlinear least squares
• 9.4. Finding the equilibrium state of a physical system by minimizing its potential energy

Chapter 10 : Signal Processing

• 10.1. Analyzing the frequency components of a signal with a Fast Fourier Transform
• 10.2. Applying a linear filter to a digital signal
• 10.3. Computing the autocorrelation of a time series

Chapter 11 : Image and Audio Processing

• 11.1. Manipulating the exposure of an image
• 11.2. Applying filters on an image
• 11.3. Segmenting an image
• 11.4. Finding points of interest in an image
• 11.5. Detecting faces in an image with OpenCV *
• 11.6. Applying digital filters to speech sounds
• 11.7. Creating a sound synthesizer in the Notebook

Chapter 12 : Deterministic Dynamical Systems

• 12.1. Plotting the bifurcation diagram of a chaotic dynamical system
• 12.2. Simulating an elementary cellular automaton
• 12.3. Simulating an ordinary differential equation with SciPy
• 12.4. Simulating a partial differential equation — reaction-diffusion systems and Turing patterns

Chapter 13 : Stochastic Dynamical Systems

• 13.1. Simulating a discrete-time Markov chain
• 13.2. Simulating a Poisson process *
• 13.3. Simulating a Brownian motion
• 13.4. Simulating a stochastic differential equation

Chapter 14 : Graphs, Geometry, and Geographic Information Systems

• 14.1. Manipulating and visualizing graphs with NetworkX *
• 14.2. Drawing flight routes with NetworkX
• 14.3. Resolving dependencies in a directed acyclic graph with a topological sort
• 14.4. Computing connected components in an image
• 14.5. Computing the Voronoi diagram of a set of points
• 14.6. Manipulating geospatial data with Cartopy
• 14.7. Creating a route planner for a road network

Chapter 15 : Symbolic and Numerical Mathematics

• 15.1. Diving into symbolic computing with SymPy
• 15.2. Solving equations and inequalities
• 15.3. Analyzing real-valued functions
• 15.4. Computing exact probabilities and manipulating random variables
• 15.5. A bit of number theory with SymPy
• 15.6. Finding a Boolean propositional formula from a truth table
• 15.7. Analyzing a nonlinear differential system — Lotka-Volterra (predator-prey) equations
• 15.8. Getting started with Sage *

Recipes marked with an asterisk * are only available in the book.

Contributing

For any comment, question, or error, please open an issue or propose a pull request.

Presentation

Python is one of the leading open source platforms for data science and numerical computing. IPython and the associated Jupyter Notebook offer efficient interfaces to Python for data analysis and interactive visualization, and they constitute an ideal gateway to the platform.

IPython Interactive Computing and Visualization Cookbook, Second Edition contains many ready-to-use, focused recipes for high-performance scientific computing and data analysis, from the latest IPython/Jupyter features to the most advanced tricks, to help you write better and faster code. You will apply these state-of-the-art methods to various real-world examples, illustrating topics in applied mathematics, scientific modeling, and machine learning.

The first part of the book covers programming techniques: code quality and reproducibility, code optimization, high- performance computing through just-in-time compilation, parallel computing, and graphics card programming. The second part tackles data science, statistics, machine learning, signal and image processing, dynamical systems, and pure and applied mathematics

cookbook-2nd's Issues

Example in 7.6 is only correct for Plate Carree

Hi,
nice work, however I believe that the KDE example in 7.6 is not quite correct, with the errors apparent if you choose any other projection than Plate Carree (x=longitude, y=latitude).

Specifically the first call to scatter:

ax.scatter(x, y, color='r', s=.5, alpha=.25)

should be something like:

ax.scatter(x, y, color='r', s=.5, alpha=.25, transform=ccrs.Geodetic())

as your source data is in lon/lat.

If you were to replace the whole cell with:

mcrs = ccrs.Mercator()
ax = plt.axes(projection=mcrs)
ax.stock_img()
ax.scatter(x, y, color='r', s=.5, alpha=.25, transform=ccrs.Geodetic())

then the data does not display correctly without the transform argument.

In the cell with:

geo = ccrs.Geodetic()
h = geo.transform_points(crs, x, y)[:, :2].T

you are actually transforming the data from Plate Carree to Geodetic, not the other way around. This, again, is only working because x=lon, y=lat.

Hope this make sense,
Regards,
Chris.

Binder not working

Hi there, it seems the Binder button for the notebooks isn't working:

Step 41/51 : RUN conda env update -p ${NB_PYTHON_PREFIX} -f "environment.yml" && conda clean --all -f -y && conda list -p ${NB_PYTHON_PREFIX}
---> Running in 9349de55aa95
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

ResolvePackageNotFound:

• pyproj==1.9.5.1=py36_0
• holoviews==1.9.0=py36h01658e4_0
• cudatoolkit==8.0=3
• util-linux==2.21=0
• proj4==4.9.3=5
• libiconv==1.15=0
• geos==3.6.2=1
• libdap4==3.19.2=1
• libpq==9.5.4=0
• libgdal==2.0.0=1

Removing intermediate container 9349de55aa95
The command '/bin/sh -c conda env update -p ${NB_PYTHON_PREFIX} -f "environment.yml" && conda clean --all -f -y && conda list -p ${NB_PYTHON_PREFIX}' returned a non-zero code: 1

Synthetizer not working

Hello, I have tried the synthetizer notebook, but it failed to work, both on my own computer and with the binder link provided. More precisely, while a call to "synth(440)" in an individual cell works as expected, clicking on the buttons, however, does not seem to trigger anything.
I made a workaround as follows.
First I created
out=widgets.Output()
Then instead of with widgets.Output() I wrote with out.
FInally in an another cell I displayed the output:
display(out)
This, however, creates a new widget each time a button is clicked. It can be circumvented by using clear_output(wait=True) after the call to 'synth', but this works only because here the duration of the sound is very short.
I'm interested in this, so if you have a better way of playing a sound (or several at the same time) without displaying unnecessary widgets, I'm all ear ;).

Nathaniel

Rolling average with stride tricks, possible erroneous timings

Hi Cyrille,
in https://ipython-books.github.io/47-implementing-an-efficient-rolling-average-algorithm-with-stride-tricks/ the bullet point no.8, benchmark of copying method, has 15 μs for copying only and 10 μs for copying and taking the mean.
Regards ፨ gb

small typing errors

Hallo some1 or two weeks ago , januari 2019I download all and am very content what you (?) did.
I found a small error in 6.1. Using matplotlib styles
matplotlib is imported as mpl but a few sections later
matplotlib was used in place of mpl ==>missing package
with a consequence of a missing () a bit later

Greets (a new beginner but very enthousiast about it)
PKHG = Peter K.H. Gragert

Error with temp_fft = sp.fftpack.fft(temp) when run in Jupyter Notebook

Hello, the code does not function for me when run in Jupyter Notebook. I replaced to_datetime with to_pydatetime. That fixed one error but now I am getting an error with temp_fft = sp.fftpack.fft(temp) it is throwing the error AttributeError: 'Series' object has no attribute 'flags'

I have updated Conda and scipy to the latest available and the error persists.
Thank you
-A

Puthon cook book!