pauldevos / fullstack-data-science-old

favorite-python-resources

A list of some favorite libraries in Python as well as specific areas of Python e.g. Data Science, Machine Learning, etc. that I've found helpful.

Popular Python APIs (with code examples)

• Module List

Popular Python ML APIs

• Skymind - Python APIs

Python Videos/Tutorials

• Raymond Hettinger - Beyond PEP 8 -- Best practices for beautiful intelligible code - PyCon 2015
• CS212 - Design of Computer Programs by Peter Norvig
• Cory Shafer Python OOP - 6 Video Series

Python Libraries

---

Data Science Workflows

• Data Science Workflows using Docker Containers(YouTube)
• How Docker Can Make You A More Effective Data Scientist

Linear Algebra

• fast.ai - Linear Algebra
• numerical-linear-algebra

Python Data Science Books

• Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow
• Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow 2 (2nd Ed)
• Python Machine Learning (2nd edition) - Sebastian Raschka
• Introduction to Artificial Neural Networks and Deep Learning - Sebastian Raschka code
• Pattern Classification - Sebastian Raschka
• Agile Data Science 2.0
• Deep Learning with Python - François Chollet
• PythonDataScienceHandbook
• practical-machine-learning-with-python
• ISLR - ISLR in Python - An Introduction to Statistical Learning (James, Witten, Hastie, Tibshirani, 2013): Python code
• Data Science from Scratch
• Data-Analysis-and-Machine-Learning-Projects
• Deep Learning Tutorials - Repository of teaching materials, code, and data for my data analysis and machine learning projects.
• TensorFlow Tutorials

---

My Favorite Tools

• Yellowbrick - Visual analysis and diagnostic tools to facilitate machine learning model selection. http://www.scikit-yb.org/
• scikit-plot - An intuitive library to add plotting functionality to scikit-learn objects.
• missingno - Visualize missing data
• Holoviews - With Holoviews, your data visualizes itself
• altair - (Github) - Declarative statistical visualization library for Python
• Pandas Styling - Mode Blog
• pandas-profiling - Create HTML profiling reports from pandas DataFrame objects

Ones to Check out

• mlens - ML-Ensemble – high performance ensemble learning http://ml-ensemble.com
• scikit-optimize - Sequential model-based optimization with a scipy.optimize interface https://scikit-optimize.github.io

---

Machine Learning Libraries

• Lime - Lime: Explaining the predictions of any machine learning classifier
• shap - A unified approach to explain the output of any machine learning model.
• eli5 - A library for debugging/inspecting machine learning classifiers and explaining their predictions
• catboost - https://github.com/catboost/catboost
• interpretable_machine_learning_with_python - Practical techniques for training interpretable ML models, explaining ML models, and debugging ML models.
• awesome-machine-learning-interpretability - A curated list of awesome machine learning interpretability resources.
• gym - A toolkit for developing and comparing reinforcement learning algorithms.
• Surprise - A Python scikit for building and analyzing recommender systems
• kubeflow - Machine Learning Toolkit for Kubernetes
• Metrics- Machine learning evaluation metrics, implemented in Python, R, Haskell, and MATLAB / Octave
• MLtest - Testing framework to simplify writing ML unit tests.
• scikit-plot - An intuitive library to add plotting functionality to scikit-learn objects.
• featuretools - An open source python framework for automated feature engineering
• Hands-on Machine Learning Model Interpretation
• Machine Learning Cheatsheet
• Learning Math for Machine Learning
• Google's Best Practices for ML Engineering
• 100 Days of ML Code
• homemade-machine-learning
• category_encoders - A library of sklearn compatible categorical variable encoders
• tensorwatch
• Snorkel - A system for quickly generating training data with weak supervision https://snorkel.org

PyTorch

• PyTorch Examples
• skorch - A scikit-learn compatible neural network library that wraps pytorch
• Skorch at Scipy 2019

Hyperparameter Tuners

• hyperband
• hyperas
• hyperopt - Distributed Asynchronous Hyperparameter Optimization in Python http://hyperopt.github.io/hyperopt

---

Notebook Tools

• papermill - Parameterize, execute, and analyze notebooks
• Notebook Kernels
• ipywebrtc - WebRTC for Jupyter notebook/lab
• ipywidgets (Notebook Example)- Interactive Widgets for the Jupyter Notebook
• itk-jupyter-widgets - Interactive Jupyter widgets to visualize images in 2D and 3D.
• jupytext - Jupyter notebooks as Markdown documents, Julia, Python or R scripts
• voila - From Jupyter notebooks to standalone web applications and dashboards - And voilà!
• nb2xls - Convert Jupyter notebook to Excel spreadsheet
• Jupyter Notebook Extensions
• StackOverflow - R & Python in One Jupyter Notebook %%R
• RISE - RISE allows you to instantly turn your Jupyter Notebooks into a slideshow. No out-of-band conversion is needed, switch from jupyter notebook to a live reveal.js-based slideshow in a single keystroke, and back.
• Tutorial: Advanced Jupyter Notebooks
• Supporting reproducibility in Jupyter through dataflow notebooks
• Explorations in reproducible analysis with Nodebook
• SoS: A polyglot notebook and workflow system for both interactive multilanguage data analysis and batch data processing
• nodebook
• jupyter-book - Create an online book with Jupyter Notebooks and Jekyll http://jupyter.org/jupyter-book

Jupyter Notebook Hubs and Extentions

• JupyterLab - JupyterLab computational environment
• jupyterhub
• Jupyter Notebook Extensions - blog - How to write a Jupyter Notebook Extension
• beakerx - Beaker Extensions for Jupyter Notebook
• nbviewer - nbconvert as a web service: Render Jupyter Notebooks as static web pages

---

AutoML

• MLBox - MLBox is a powerful Automated Machine Learning python library. https://mlbox.readthedocs.io/en/latest/
• tpot - A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming. http://epistasislab.github.io/tpot/
• auto-sklearn - Automated Machine Learning with scikit-learn https://automl.github.io/auto-sklearn
• H20ai - Tutorials and training material for the H2O Machine Learning Platform
• adanet - Fast and flexible AutoML with learning guarantees. https://adanet.readthedocs.io

Statistics

• pgmpy - Notebook tutorials - Python Library for Probabilistic Graphical Models
• Intro2Stats (Tutorial)
• Thinks Stats 2nd ed

Bayesian Statistics

• Bayesian-Modelling-in-Python

• PyMC3 - Probabilistic Programming in Python: Bayesian Modeling and Probabilistic Machine Learning

• PyMC3 Resources/Tutorials - PyMC3 educational resources - (Textbook)

• arviz - Exploratory analysis of Bayesian models with Python https://arviz-devs.github.io/arviz/

• Bayesian Podcast

• discourse.pymc.io

• Think Bayes

---

Data Visualization -- Inspiration and Answers

• Python Graph Gallery
• Top 50 matplotlib Visualizations – The Master Plots
• From Data to Viz - Graphing Decisions - code in R
• The Art of Effective Visualization of Multi-dimensional Data
• Policy Viz - DataViz Books
• Python Plotting for Exploratory Data Analysis
• Observable - D3 and Other Data Vizualizations
• Our World in Data (Visualizations)
• Open Data Science - Data Visualization – How to Pick the Right Chart Type?
• Data Viz Project | Collection of data visualizations to get inspired and finding the right type.

Data Visualization

• Yellowbrick - Visual analysis and diagnostic tools to facilitate machine learning model selection. http://www.scikit-yb.org/
• scikit-plot - An intuitive library to add plotting functionality to scikit-learn objects.
• ipyvolume - 3d plotting for Python in the Jupyter notebook based on IPython widgets using WebGL
• Dash - Written on top of Flask, Plotly.js, and React.js, Dash can be used for highly custom user interfaces in Python
• Chartify - Same interface for plots with a goal to make it easy to use Bokeh plots.
• bqplot - 2-D plotting library for Project Jupyter
• ipyleaflet - A Jupyter / Leaflet bridge enabling interactive maps in the Jupyter notebook.
• pythreejs - A Python / ThreeJS bridge utilizing the Jupyter widget infrastructure.
• gmaps - Google Maps For Jupyter Notebooks
• vaex - Lazy Out-of-Core DataFrames for Python. Visualize a billion rows per second on a single computer.
• scattertext - Beautiful visualizations of how language differs among document types.
• missingno - Visualize missing data
• pygal - Python svg Graph plotting Library
• geoplotlib - python toolbox for visualizing geographical data and making maps
• gleam - Creating interactive visualizations with Python
• geonotebook - A Jupyter notebook extension for geospatial visualization and analysis
• Bokeh Tutorial (Notebooks)
• Bokeh
• Responsive Bar Charts with Bokeh, Flask and Python 3
• Dash - Analytical Web Apps for Python. No JavaScript Required.
• Creating Interactive Visualizations with Plotly’s Dash Framework
• Superset - Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application
• NetworkX - NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.
• Seaborn - Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.
• ggpy - ggplot for Python
• Matplotlib - matplotlib: plotting with Python
• Effectively Using Matplotlib
• Plotly - An open-source, interactive graphing library for Python
• Generate HTML reports with D3 graphs using Python, Plotly, and Pandas
• AnatomyOfMatplotlib (Tutorial) - Anatomy of Matplotlib -- tutorial developed for the SciPy conference
• pyLDAvis - Python library for interactive topic model visualization. Port of the R LDAvis package
• DataShader - uses Dask and Numba to visualize huge data sets
• Graph-Tool - Graph-tool is an efficient Python module for manipulation and statistical analysis of graphs (a.k.a. networks)
• Vispy
• Panel - A high-level Python toolkit for composing widgets and plots https://panel.pyviz.org
• jupyter-matplotlib - Matplotlib Jupyter Extension
• ipysheet - Jupyter handsontable integration
• ipywebrtc - WebRTC for Jupyter notebook/lab
• weasyprint - WeasyPrint converts web documents (HTML with CSS, SVG, …) to PDF. https://weasyprint.org/ (Creating PDF Reports with Pandas, Jinja and WeasyPrint)
• PySal - PySAL: Python Spatial Analysis Library http://pysal.org

Data

• Our World in Data - data visualizations of many things around the world

---

Monte Carlo Simulations

• PBP - Monte Carlo Simulation with Python - (Code)
• Monte Carlo simulation in Python

Data Pipeline Tools

• Pandas - Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
• Modin - Modin: Speed up your Pandas workflows by changing a single line of code
• Pandas-Profiling - Create HTML profiling reports from pandas DataFrame objects
• Dask - Parallel computing with task scheduling
• ray - A system for parallel and distributed Python that unifies the ML ecosystem.
• Spark - PySpark API
• Optimus - Agile Data Science Workflows made easy with Pyspark https://hioptimus.com
• pypeln- Concurrent data pipelines made easy
• smart_open - Utils for streaming large files (S3, HDFS, gzip, bz2...)
• Blaze - NumPy and Pandas interface to Big Data
• Faker - Faker is a Python package that generates fake data for you.
• Kedro - A Python library for building robust production-ready data and analytics pipelines

---

Time Series

• Prophet - A Facebook Time Series Analysis library
• PyFlux - Time series analysis library with flexible range of modelling and inference options.

---

NLP/Text Manipulation

• spaCy - Industrial-strength Natural Language Processing (NLP) with Python and Cython
• FlashText - This module can be used to replace keywords in sentences or extract keywords from sentences.
• textacy - NLP, before and after spaCy
• FuzzyWuzzy - Fuzzy string matching in Python
• [re] (regex)
• pampy
• textblob - Simple, Pythonic, text processing--Sentiment analysis, part-of-speech tagging, noun phrase extraction, translation, and more.
• nlp-text-mining-working-examples
• Pattern - Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization
• Feature Engineering Text Data - Traditional Strategies
• Feature Engineering Text Data - Advanced Deep Learning Strategies
• NLP Strategy I - Processing and Understanding Text
• Practical Text Classification With Python and Keras
• Regex tutorial — A quick cheatsheet by examples
• spaCy - Industrial-strength Natural Language Processing (NLP) with Python and Cython https://spacy.io
• xlnet - XLNet: Generalized Autoregressive Pretraining for Language Understanding
• xlnet-Pytorch - Simple XLNet implementation with Pytorch Wrapper https://arxiv.org/pdf/1906.08237.pdf

---

Deep Learning Tools

• Pyro - Pyro is a flexible, scalable deep probabilistic programming library built on PyTorch
• allennlp - An open-source NLP research library, built on PyTorch.
• PyTorch-NLP - Supporting Rapid Prototyping with a Toolkit (incl. Datasets and Neural Network Layers)
• OpenCV-Python - Open Source Computer Vision Library for Python
• Screenshot-to-code - A neural network that transforms a design mock-up into a static website
• TensorFlow Models Examples
• fastai - The fastai deep learning library, plus lessons and and tutorials
• Keras
• TensorFlow
• keras-tuner - Hyperparameter tuning for humans
• Visdom - A flexible tool for creating, organizing, and sharing visualizations of live, rich data. Supports Torch and Numpy.
• talos - Hyperparameter Optimization for Keras Models
• keras-contrib

Recommender Systems

• list_of_recommender_systems
• Amazon-Product-Recommender-System - Sentiment analysis on Amazon Review Dataset available at http://snap.stanford.edu/data/web-Amazon.html
• Building a Recommendation System Using Neural Network Embeddings
• Neural Network Embeddings Explained

Web Scraping Tools

• BeautifulSoup
• requests
• MechanicalSoup - A Python library for automating interaction with websites.

SDLC Tools

• yapf - yet another python formatter
• black - The uncompromising Python code formatter
• precommit (article) - A framework for managing and maintaining multi-language pre-commit hooks. (Black + Flake8)

Others

• wget - free utility for non-interactive downloading files from the web
• pendulum - Python datetime manipulation made easy
• python-dotenv - Get and set values in your .env file in local and production servers.
• sqlparse - sql parsing tool
• credstash - A little utility for managing credentials in the cloud

Database Connectors

• graphene - GraphQL framework for Python

Pub-Sub, Message Queues, Streaming

• kq - Kafka-based Job Queue for Python
• sockets
• pykafka - Apache Kafka client for Python; high-level & low-level consumer/producer, with great performance.
• awesome-kafka - A list about Apache Kafka
• How to Stream Text Data from Twitch with Sockets in Python

Web Frameworks Some Benchmarks

• Pyramid - A Python web framework https://trypyramid.com/
• sanic - Async Python 3.5+ web server that's written to go fast
• Tornado - Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed.
• Falcon - Falcon is a bare-metal Python web API framework for building high-performance microservices, app backends, and higher-level frameworks. Docs
• Vibora - Fast, asynchronous and elegant Python web framework.
• japronto - Screaming-fast Python 3.5+ HTTP toolkit integrated with pipelining HTTP server based on uvloop and picohttpparser.
• aiohttp - Asynchronous HTTP client/server framework for asyncio and Python https://docs.aiohttp.org
• fastapi, Docs - FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.6+ based on standard Python type hints.

Networking Tools

• uvloop (article) - Ultra fast asyncio event loop (faster than NodeJS, close to Go speed)

Flask Tools

• Flask-AppBuilder - Simple and rapid application development framework, built on top of Flask. includes detailed security, auto CRUD generation for your models, google charts and much more.
• Awesome-Flask - A curated list of awesome Flask resources and plugins

Static File Generators

• Nikola - A static website and blog generator
• Pelican - Static site generator that supports Markdown and reST syntax. Powered by Python.

PDF Converters

• python-pdfkit - Wkhtmltopdf python wrapper to convert html to pdf
• pdftabextract - A set of tools for extracting tables from PDF files helping to do data mining on (OCR-processed) scanned documents.

Documentation Libraries

• Sphinx
• pweave - Pweave is a scientific report generator and a literate programming tool for Python. It can capture the results and plots from data analysis and works well with numpy, scipy and matplotlib.

---

D3

• Hitchhiker's Guide to D3.js
• D3 Charts and Chartbuilder
• D3 Tutorials
• D3 API Reference
• Awesome-D3
• Data Visualization with D3.js, a FreeCodeCamp course

APIs

• Tweepy

Hacking, PCAPs, and Network Analysis

• Sublist3r
• scapy
• knock - subdomain scanner
• scapy-http
• dpkt
• kamene
• pcapy - Pcapy is a Python extension module that interfaces with the libpcap packet capture library.
• pyshark - Python wrapper for tshark, allowing python packet parsing using wireshark dissectors
• PyPCAPKit - Python multi-engine PCAP analyse kit.
• fsociety - fsociety Hacking Tools Pack – A Penetration Testing Framework
• PayloadsAllTheThings - A list of useful payloads and bypass for Web Application Security and Pentest/CTF

GUIs

• PySimpleGUI

Repeatable Python Workflows in Notebooks

• Building a Repeatable Data Analysis Process with Jupyter Notebooks

Prototyping Projects

• Learn to Build Machine Learning Services, Prototype Real Applications, and Deploy your Work to Users
• Combining D3 with Kedion: Graduating from Toy Visuals to Real Applications
• Learning How to Build a Web Application

Machine Learning - Interesting Repos

• Siraj Raval https://github.com/rushter/MLAlgorithms https://github.com/WillKoehrsen/Data-Analysis https://github.com/WillKoehrsen/machine-learning-project-walkthrough https://github.com/Avik-Jain/100-Days-Of-ML-Code https://github.com/llSourcell/Learn_Data_Science_in_3_Months https://github.com/llSourcell/Learn_Machine_Learning_in_3_Months https://github.com/llSourcell/100_Days_of_ML_Code https://github.com/ZuzooVn/machine-learning-for-software-engineers https://github.com/llSourcell/Learn_Deep_Learning_in_6_Weeks https://github.com/Spandan-Madan/DeepLearningProject
• Machine Learning from Scratch
• Jupyter notebooks from the scikit-learn video series - Justin Markham
• fast.ai - The fastai deep learning library, plus lessons and and tutorials http://docs.fast.ai
• Coursera ML - Andrew Ng in Python
• Think Python 2nd ed
• Thinks Stats 2nd ed
• Think Bayes
• Make Your Own Neural Network
• Learning AI if you Suck at Math
• Neural Networks and Deep Learning
• Introduction to Statistics With Python

ML Articles

• How To Create Data Products That Are Magical Using Sequence-to-Sequence Models
• How To Create Natural Language Semantic Search For Arbitrary Objects With Deep Learning
• Machine Learning for Everyone

Docker Articles

• Docker articles by Docker

Git Commands

• Git-Commands

Codelabs

• https://codelabs.developers.google.com/?cat=tensorflow

Awesome Links

• Awesome-BigData
• Awesome-Hacking-Resources
• awesome-web-hacking
• awesome-data-engineering
• awesome-elasticsearch
• awesome-datascience
• awesome-flask

NBA Shot Charts

• nbashots - NBA shot charts using matplotlib, seaborn, and bokeh.
• nba-team-shot-charts

---

Courses

Deep Learning Courses

• Fast.ai

---

More Random Packages

• Data Science Blogs
• Drag and Drop JS
• Projects - Project Solutions
• project-based-learning#python

Notebook tools -- Older, less maintained

• ipyaggrid (Features) (Article) - Using ag-Grid in Jupyter notebooks
• PixieDust (YouTube Example) (DeBugger article) - PixieDust 1.0 is here (blog) - Easy Data Visualizer, Debugger, etc for Notebooks
• nbdime - Tools for diffing and merging of Jupyter notebooks
• nteract - The interactive computing suite for you!
• Sparkmagic - Jupyter magics and kernels for working with remote Spark clusters
• PayPal Notebooks, powered by Jupyter: Enabling the next generation of data scientists at scale
• PayPal Notebook Extensions
• nbtutor - Visualize Python code execution (line-by-line) in Jupyter Notebook cells.
• py_d3 - D3 block magic for Jupyter notebook
• Jupyter Notebook Formatting Extensions
• Jupyter Notebooks Dashboards Extension
• JupyterLab-toc - Table of Contents Extension
• Jupyter Extension Tricks
• Bringing the best out of Jupyter Notebooks for Data Science
• Airbnb - Knowledge Repo - tag and share analysis
• Advanced Jupyter Notebooks: A Tutorial
• imgkit - Wkhtmltoimage python wrapper to convert html to image

Old...