haixu-qin / minigrad

A Mini Gradient Descent library.

MiniGrad

- a Mini Auto Differentiation / Gradient Descent library for teaching purposes.
Ref: micrograd, and other online webs.

Probably one of the lightest NN library online.
much faster for small datasets.
Pros: small space, fast runtime. easy to use for small datasets.
Cons: not optimized in terms of big datasets.

Example

Prereqs: pip3 install numpy
Run: python3 example.py

Example Output: [0, 1, 1, 0]

Math Explanation

Math Explanation for .backward() (for educational purposes).
It could be found there.

Math Explanation for .backward_opt() (for better and faster convergence).
It could be found there.

Math Explanation for multi-D y (./iris/minigrad_iris.py).
It could be found there.

Optimizers.

Optimizer is used by the library by default.
RAdam (Rectified Adam), which introduces a rectification term in addition to the popular standard Adam optimizer (which uses Momentum and RMSprop and a bias-correction mechanism). It performs well for small datasets.

Multi-D Y

For additional support for multi-D y, check out /iris. I seperated them out so it's easier to study the basics bc it's more of a tutorial.