mikeinnes / diff-zoo

Differentiation for Hackers

Differentiation for Hackers

The goal of this handbook is to demystify algorithmic differentiation, the tool that underlies modern machine learning. It begins with a calculus-101 style understanding and gradually extends this to build toy implementations of systems similar to PyTorch and TensorFlow. I have tried to clarify the relationships between every kind of differentiation I can think of – including forward and reverse, symbolic, numeric, tracing and source transformation. Where typical real-word ADs are mired in implementation details, these implementations are designed to be coherent enough that the real, fundamental differences – of which there are surprisingly few – become obvious.

The intro notebook is recommended to start with, but otherwise notebooks do not have a fixed order.

• Intro – explains the basics, beginning with a simple symbolic differentiation routine.
• Back & Forth – discusses the difference between forward and reverse mode AD.
• Forward – discusses forward-mode AD and its relationship to symbolic and numerical differentiation.
• Tracing – discusses tracing-based implementations of reverse mode, as used by TensorFlow and PyTorch.
• Reverse – discusses a more powerful reverse mode based on source transformation (not complete).

If you want to run the notebooks locally, they can be built by running the src/notebooks.jl script using Julia. They should appear inside a /notebooks folder. Alternatively, you can run through the scripts in Juno.

diff-zoo's Issues

Typo in intro

The first sentence reads:

These notebooks are an exploration of various approaches to analytical differentiation.

From what I can tell, "analytical" should be "algorithmic".

tracing.jl notebook will not build

┌ Error: error when executing notebook based on input file: `<path>\diff-zoo\src\tracing.jl`
└ @ Literate <path>\.julia\packages\Literate\3qT7A\src\Literate.jl:555
ERROR: LoadError: LoadError: syntax: unexpected semicolon in tuple around string:1
in expression starting at string:1
when executing the following code block

@eval dcube(x) = $(Expr(dy))

dcube(5)

If I comment out these lines, then the next lines error too.
I tested with Julia 1.6 and 1.10 and with Literate v2.2.0 and v2.16.0. All had the same error.
There is also a Pkg error updating dependency versions unless name = "diff-zoo" is removed from the Project.toml.

Division Formulae are Wrong: Reverse Diff

diff-zoo/src/utils.jl

Line 130 in 9f48c85

d(a, push!(w, mulm(Δ, b)))

It should be divm (not mulm).

Hope can finish reverse notebook

Hi , awesome tutorial !!! Hope can finish reverse notebook, very interesting in theory behind Zygote.jl

Add source-to-source reverse notebook

Just a nudge to please add the source-to-source reverse mode notebook.

Minor details and personal feedback.

First I want to thanks you because this is perfectly done!
It help me a lot to understand because your truly start from the basics with very high quality explanation.

-LaTeX rendering bug after out[31] (in the intro part).

-In[20] (in the forward part) could be write with LaTeX in the text to help fast readers.
Also this section "Perturbation Confusion" could have a conclusion about solutions or the lack of solution to fix this problem.

Division Formulae are Wrong : Dual Numbers

Thank you for this extremely wonderful tutorial. I learnt a lot.
Also please mention that they need to read backandforth.ipynb after the intro.ipynb. It makes most sense that way. And it took me a while to figure that out.

And now the bug...

struct Dual{T<:Real} <: Real
    x::T
    y::T
end
Base.show(io::IO, d::Dual) = println(io, d.x, d.y≥0 ? "+" : "", d.y, "ϵ")

import Base: +, -, *, /
a::Dual + b::Dual = Dual(a.x + b.x, a.y + b.y)
a::Dual * b::Dual = Dual(a.x * b.x, b.x * a.y + a.x * b.y)
a::Dual / b::Dual = Dual(a.x / b.x,(b.x * a.y - a.x * b.y)/b.x^2)

Base.convert(::Type{Dual{T}}, a::Dual) where T = Dual(convert(T, a.x), convert(T, a.y))
Base.convert(::Type{Dual{T}}, a::Real) where T = Dual(convert(T, a), zero(T))
Base.promote_rule(::Type{Dual{T}}, ::Type{R}) where {T,R} = Dual{promote_type(T,R)}

Now let us try a simple function. I am also giving a numerical derivative for reference.

f(x) = x/(1+x*x)
ndf(x, ϵ=√eps()) = (f(x+ϵ)-f(x-ϵ))/2ϵ
f(7), ndf(7)

(0.14, -0.01919999998062849)

Now let us try with Dual numbers

f(Dual(7., 1.))

0.14-0.0192ϵ

Works!

But with your definition of division,

a::Dual / b::Dual = Dual(a.x * b.x, b.x * a.y - a.x * b.y)
f(Dual(7., 1.))

350.0-48.0ϵ

It is obviously wrong. Also you can see that my formula for division matches with your high school text book.