epogrebnyak / econometrics-navigator

Review econometrics concepts with code examples

Hi I'm Evgeniy Pogrebnyak

Work

• Finec MGIMO, where we teach economics, management and business IT.
• Previously VEB, ICSS and forecast.ru.

Publications

• Google Scholar
• 
• elibrary ИНП РАН, МГИМО
• Scopus

Datasets

I maintain several open datasets and codebooks:

Name	Content	Years	Github
boo	Russian firms annual financial statements	2012-2018
weo	World Economic Outlook releases as pandas dataframes	2007+
ssg	Static site generators popularity on Github	2021

Read more here

Publications

• elibrary.ru author profile
• Google scholar

Topics: power markets, economics of automotive industry, industrial and competition policies, exchange rates, sustainable development goals.

Thesis: Policy parameters and regulatory controls for Russian competitive electricity market

Also published: dictionary of Russian business slang.

Other links

• My bio in Russian here

econometrics-navigator's Issues

Assimilate page form another repo

• primitives repo
• inference book
• ODS repo

• https://github.com/Yorko/mlcourse.ai
• https://github.com/astorfi/Deep-Learning-World

ML Book

https://web.stanford.edu/~hastie/Papers/ESLII.pdf""

Data Science

https://www.stat.berkeley.edu/~aldous/134/grinstead.pdf

https://conferences.oreilly.com/strata/strata-eu/public/schedule/detail/64616

Ng: https://www.coursera.org/learn/machine-learning

https://www.edx.org/course/data-science-essentials

https://academy.microsoft.com/en-us/tracks/data-science

https://aischool.microsoft.com/learning-paths/4Hd8IqZmk0eegO4aiGKkOO/modules/6TAKWap1mg8sg8GcK8Iq8w

https://jakevdp.github.io/PythonDataScienceHandbook/index.html

Probabilistic programming

• http://probabilistic-programming.org/wiki/Home
• http://jamesrobertlloyd.com/talks/prob-prog-intro.pdf
• https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
• https://github.com/pymc-devs/pymc3
• https://peerj.com/articles/cs-55.pdf

This question leads to below:

• messy leature notes
• book with no kindle version

Suggestion:

• Bishop

ML/DL

Books and courses

• Deep Learning Book
• fast.ai
• Andrew Ng at Coursera and at deeplearning.ai

Videos

• Sberbank lecture on ML limitations (Vetrov, youtube URL at J5HOjC4Xn_Y)

Journals

• gallery of ML papers
• open-source ML research journal

Speakers

• Ostrov 2018 DL speakers were Denis Volkhonskiy and
  Victor Kulikov
• Also a small course by Atcold on PyTorch

Build your own neural netowrk

• PyTorch numpy neural network tutorial

Blockchain

• https://bitcoin.org/bitcoin.pdf
• https://medium.com/crypto-currently/lets-build-the-tiniest-blockchain-e70965a248b
• https://hackernoon.com/learn-blockchains-by-building-one-117428612f46
• http://ecomunsing.com/build-your-own-blockchain

Comments

Bishop at Microsoft: https://www.microsoft.com/en-us/research/people/cmbishop/

Review article: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/05/Bishop-MBML-2012.pdf

Like fast.ai: https://www.deeplearning.ai/

Three giants survey in Nature taken from (https://github.com/floodsung/Deep-Learning-Papers-Reading-Roadmap) + The Bible: http://www.deeplearningbook.org

More

Introduction to ML (Nilsson/Stanford), Gaussian Process for ML, Introduction to ML (Alpaydin),
Information Theory Inference and Learning Algorithms (very useful book), Machine Learning (Mitchell),
Pattern Recognition and Machine Learning (standard ML course book at Edinburgh and various
Unis but relatively a heavy reading with math),

• Information Theory Inference and Learning Algorithms
• Machine Learning (Mitchell)
• 1996
• http://www.gaussianprocess.org/gpml/
• Alpaydin
• Bishop

This question leads to below:

• messy leature notes
• book with no kindle version

Writing section - Scott Hotaling

https://twitter.com/MtnScience/status/1272904835600392192

Ryan Adams + Gelman et al links

Ryan Adams 
CS281: Advanced Machine Learning
Harvard University, Fall 2013
https://seas.harvard.edu/courses/cs281/

HT @rahuldave

Great outline, see also course textbooks in specific in syllabus:

The following book is required for the course:

• Machine Learning: A Probabilistic PerspectiveKevin P. Murphy, MIT Press, 2012.
  This is a verynew book that covers a wide set of important topics. As the book is fresh and comprehensive,there are still quite a few errors. We will try to maintain lists of errata as they are discovered.
  The following book is strongly recommended, but not required:

• Pattern Recognition and Machine LearningChristopher M. Bishop, Springer, 2006. An excellentand affordable book on machine learning, with a Bayesian focus. It covers fewer topics thanthe Murphy book, but goes into greater depth on many of them and you may find that youprefer Bishop’s exposition.

These are other (free online!) books on machine learning and related topics that you may findhelpful, but that are completely optional.:

Information Theory, Inference, and Learning AlgorithmsDavid J.C. MacKay, Cambridge Uni-versity Press, 2003. Freely available online athttp://www.inference.phy.cam.ac.uk/mackay/itila/. A very well-written book with excellent explanations of many ma-chine learning topics.

Bayesian Reasoning and Machine LearningDavid Barber, Cambridge University Press, 2012. Freelyavailable online athttp://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online.3

The Elements of Statistical LearningTrevor Hastie, Robert Tibshirani, and Jerome Friedman, Springer,2009. Freely available online athttp://www-stat.stanford.edu/ ̃tibs/ElemStatLearn/.

These are books on some specialized topics that you may find useful:

Gaussian Processes for Machine Learning. Carl Edward Rasmussen and Christopher K.I. Williams,MIT Press, 2006. Freely available online athttp://www.gaussianprocess.org/gpml/.

Non-Uniform Random Variate GenerationLuc Devroye, Springer-Verlag, 1986. Freely availableonline athttp://luc.devroye.org/rnbookindex.html.

Probabilistic Graphical Models: Principles and TechniquesDaphne Koller and Nir Friedman,MIT Press, 2009.

Numerical Optimization Jorge Nocedal and Stephen J. Wright, Springer, 2006.

Bayesian Data Analysis. Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. CRC, 2013.

NOTE: soon a new edition, codebook by @avehtari here,
book release page: https://www.cambridge.org/core/books/regression-and-other-stories/DD20DD6C9057118581076E54E40C372C (likely comes sooner than in September)

Elements of Information Theory Thomas M. Cover and Joy A. Thomas, Wiley, 1991.

Monte Carlo Statistical Methods Christian P. Robert and George Casella, Springer, 2005.

to add

https://twitter.com/zerdeve/status/1193389994208923648

Berna Devezer @zerdeve

I tweeted about Meng’s (2018) Statistical Paradises and Paradoxes paper in Annals of Applied Statistics before but I want to highlight just how much I love the writing in the introduction. This is great stuff. I’ll quote some more below.

---

https://twitter.com/cubic_logic/status/1192902639526273025

(λ --- --. .. -.-.)³ @cubic_logic

Non-randomly missing data: why weights won’t solve your survey problems and you need to think generatively.

https://statmodeling.stat.columbia.edu/2019/10/29/non-random-missing-data-weights-generative-model/

---

Stan Hurn et al - Financial econometrics

https://global.oup.com/academic/product/financial-econometric-modeling-9780190857127?lang=en&cc=us
https://fxdiebold.blogspot.com/2020/07/outstanding-new-financial-econometrics.html

use invoke to organise build/push/clean commands

RemovedInSphinx30Warning in sphinx_rtd_theme

d:\anaconda3\lib\site-packages\m2r.py:652: RemovedInSphinx30Warning: app.add_source_parser() does not support suffix argument. Use app.add_source_suffix() instead.
  app.add_source_parser('.md', M2RParser)
d:\anaconda3\lib\site-packages\sphinx_rtd_theme\search.html:20: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead.
  {{ super() }}

Links to readthedocs/sphinx_rtd_theme#739

separate markdown source and builds

must use gh-pages branch and change push.bat

econometrics-navigator/push.bat

Lines 1 to 6 in c939a54

	REM TODO: switch branch, and markdown and add to gh-pages
	@echo off
	if NOT [%1]==[] (
	git commit -am%1
	git push
	)

Make pdf of documentation with latex

Some steps at https://habr.com/ru/post/328182/

meta guide for teaching / self-study

Example: https://metacademy.org/roadmaps/rgrosse/bayesian_machine_learning and for example the
"latent dirichlet allocation" linked from it. But i think their needs to be a context to it.
via @rahuldave

Opinion:

• several articles from metacademy.org are good, but overall listing of articles need a raodmap of its own
• creating content is hard, need set motivation for learners that fuels interest trough the topic (context above for non-programmers)
• our Concepts section in trics.me needs rework - we could not attain good coverage of the topics and critical mass of arcticles, in a way we are in metaacademy situation there.

Economics and computation

Check Computational methods for economists at https://www.sas.upenn.edu/~jesusfv/teaching.html

This set of lecture notes has been prepared for my class on computational methods.

Lecture 1: High-performance computing in economics.

Lecture 2: Software engineering.

Lecture 3: OS and basic utilities.

Lecture 4: Concepts on programming languages.

Lecture 5: Scientific computing languages.

Lecture 6: Coding tools.

Lecture 7: Programming paradigms.

Lecture 8: The elements of programming style.

Lecture 9: Data handling.

Lecture 10: Web scrapping.

Lecture 11: Paralellization.

Lecture 12: Numerical differentiation and integration.

Lecture 13: Optimization.

Lecture 14: Value function iteration.

Lecture 15: Computational complexity.

Lecture 16: Nonlinear methods.

Lecture 17: Projection methods.

Lecture 18: Perturbation methods I, basic results.

Lecture 19: Perturbation methods II, general case.

Lecture 20: Perturbation methods III, change of variables.

Lecture 21: Perturbation methods IV, perturbing the value function.

Lecture 22: Perturbation methods V, pruning

Lecture 23: Appendix on linearization.

Extra material:

Chapter on software engineering for economists.

Chapter on Unix.

Chapter on Git.

Chapter on Make.

Chapter on notebooks, markdown, and Pandoc.

Chapter on Julia. Now for Julia 1.1! Also, check my script for a 4-hour tutorial on Julia here and a good cheat sheet here.

A Practical Guide to Parallelization in Economics.

My github page: here.

The github page on parallelization: here.

Some codes:

A basic RBC model.

An RBC model with stochastic volatility.

An RBC with EZ preferences, Taylor rule, and yield curve.

An RBC computed with Chebyshev polynomials.

An example of memory locality. 

D Hendry - Intro to macroeconometrics

https://www.timberlake.co.uk/intromacroeconometrics

Add MLE tutorial

https://twitter.com/johnmyleswhite/status/1264355256974168067

https://trics.me/topics/max-likelihood.html

Accomodate example

# https://github.com/arturomp/coursera-machine-learning-in-python/blob/master/mlclass-ex2-004/ex2.pdf
# https://stackoverflow.com/questions/52286971/regularized-logistic-regression-in-python-andrew-ng-course?noredirect=1#comment91524735_52286971

import numpy as np
import pandas as pd
import scipy.optimize as op
import seaborn as sns


URL = ('https://raw.githubusercontent.com/TheGirlWhiteWithBandages/'
       'Machine-Learning-Algorithms/master/Logistic%20Regression/ex2data2.txt')


# Read the data and give it labels
data = pd.read_csv(URL, header=None, names=['Test1', 'Test2', 'Accepted'])


# plot dataset
# https://stackoverflow.com/questions/21654635/scatter-plots-in-pandas-pyplot-how-to-plot-by-category
df = data
df.columns = ['x1', 'x2', 'y']
# center x1 and  x2 around mean
df.loc[:,['x1', 'x2']] -= df.mean()[['x1', 'x2']]    
sns.pairplot(x_vars=["x1"], y_vars=["x2"], data=df, hue="y", size=5) 
# - when deviations for center are small, an item is accepted 
# - at some measurements there both accepted and rejected items

# Separate the features to make it fit into the mapFeature function
X1 = data['Test1'].values.T
X2 = data['Test2'].values.T
y = np.matrix(data['Accepted'].values).T

 


def mapFeature(X1, X2, degree = 6):
    
# https://github.com/arturomp/coursera-machine-learning-in-python/blob/master/mlclass-ex2-004/mlclass-ex2/mapFeature.py

# MAPFEATURE Feature mapping function to polynomial features
#
#   MAPFEATURE(X1, X2) maps the two input features
#   to quadratic features used in the regularization exercise.
#
#   Returns a new feature array with more features, comprising of 
#   X1, X2, X1.^2, X2.^2, X1*X2, X1*X2.^2, etc..
#   for a total of 1 + 2 + ... + (degree+1) = ((degree+1) * (degree+2)) / 2 columns
#
#   Inputs X1, X2 must be the same size
    dim1 = X1.shape[0]
    dim2 = sum(range(degree + 2)) # could also use ((degree+1) * (degree+2)) / 2 instead of sum
    out = np.ones([dim1, dim2]) 
    curr_column = 1
    for i in range(1, degree + 1):
        for j in range(i+1):
            out[:,curr_column] = np.power(X1,i-j) * np.power(X2,j)
            curr_column += 1
    return out


# Separate the data into training and target
X = mapFeature(X1, X2)
m, n = X.shape
# initialize theta
theta = np.matrix(np.zeros(n))

#Initialize the learningRate(sigma)
learningRate = 1


# Define the Sigmoid Function (Output between 0 and 1)
def sigmoid(z):
    return 1 / (1 + np.exp(-z))


def cost(theta, X, y, learningRate):
    # This is require to make the optimize function work
    theta = theta.reshape(-1, 1)
    error = sigmoid(X @ theta)
    first = np.multiply(-y, np.log(error))
    second = np.multiply(1 - y, np.log(1 - error))
    return np.sum((first - second)) / m + (learningRate * np.sum(np.power(theta, 2)) / 2 * m)



    
# Define the gradient of the cost function
def gradient(theta, X, y, learningRate):
    # This is require to make the optimize function work
    theta = theta.reshape(-1, 1)
    error = sigmoid(X @ theta)
    grad =  (X.T @ (error - y)) / m + ((learningRate * theta) / m)
    grad_no = (X.T @ (error - y)) / m
    grad[0] = grad_no[0]
    return grad


result = op.minimize(fun=cost, x0=theta, args=(X, y, learningRate), method='TNC', jac=gradient)
opt_theta = np.matrix(result.x)


def predict(theta, X):
    sigValue = sigmoid(X @ theta.T)
    p = sigValue >= 0.5
    return p


p = predict(opt_theta, X)
print('Train Accuracy: {:f}'.format(np.mean(p == y) * 100))