Part of this repository is based on predicting-poverty.
The data and code in this repository allows users to fully reproduce the results appearing in the main text of the paper Predicting poverty using daylight satelliteimagery
The code works on Python 3.7 and R 3.5.1
- data: Input and output data stored here
- figures: Notebooks used to generate Figs. 3-5
- scripts: Scripts used to process data
- model: Store parameters for trained convolutional neural network
R
- R.utils
- magrittr
- foreign
- raster
- readstata13
- plyr
- RColorBrewer
- sp
- lattice
- ggplot2
- grid
- gridExtra
The user can run the following command to automatically install the R packages
install.packages(c('R.utils', 'magrittr', 'foreign', 'raster', 'readstata13', 'plyr', 'RColorBrewer', 'sp', 'lattice', 'ggplot2', 'grid', 'gridExtra'), dependencies = T)
Python
- NumPy
- Pandas
- SciPy
- scikit-learn
- Seaborn
- Geospatial Data Abstraction Library (GDAL)
Due to data access agreements, users need to independently download data files from the World Bank's Living Standards Measurement Surveys websites. These data source requires the user to fill in a Data User Agreement form.
For all data processing scripts, the user needs to set the working directory to the repository scripts
folder.
-
Download LSMS data
- Visit the host website for the World Bank's LSMS-ISA data:
- Download into data/input/LSMS the files corresponding to the following country-years:
- Malawi 2016
-
Run the following files in the script folder
- DowndloadPublicData.R (run from root folder)
- ProcessSurveyData.R (run from root folder)
- save_survey_data.py
-
Download the parameters of the trained CNN model here and save in the model directory.
-
Generate candidate locations to download using
get_image_download_locations.py
. This will generate locations meant to download 1x1 km RGB satellite images of size 400x400 pixels. Locations for 121 images in a 10x10 km area around the cluster is generated. The result of running this is a file for each country, for each dataset namedcandidate_download_locs.txt
, in the following format for every line:[image_lat] [image_long] [cluster_lat] [cluster_long]
For example, a line in this file may be
4.163456 6.083456 4.123456 6.123456
Note that this requires GDAL and previously running
DownloadPublicData.R
. -
Download all images generated in
candidate_download_locs.csv
using Satellite-Downloader. The images should be stored in `data/input/Daylights/COUNTRY/ Attention: A google clould platform registration is needed! -
Download high resolution nightlight data here from 2016 (yearly) (ony tile 5 is needed) and save the
vcm-orm-ntl
version indata/input/Nightlights/2016/south_middle.tif
-
Run
process_sat_images.py
. This splits the square kilometer images to 500m x 500m images to fit the resolution of the nightlight data and then saves the images in../data/output/Daylights/malawi/split_complete
. This includes the folderstrain
,test
&val
with subfoldersNO_LIGHT
,LOW_LIGHT
&HIGH_LIGHT
which are the three classes for the first step in our neural network. This set is highley unbalanced but is fully needed for 6. For learning purposes most of theNO_LIGHT
imagery was removed. -
Run
sat2poverty.py
to generate a similar data structure for below poverty line and above poverty line. -
Training and evaluation can be done by using the jupyter notebooks
vgg_nightlights.ipynb
andvgg_poverty.ipynb
.- There is also a copy available for training on Google Colab.