one-acre-fund / arc2weather

R package for systematically downloading available arc2 weather .tif files for east Africa

Africa Rainfall Climatology weather data package

Work in progress Check with Matt Lowes if you have questions!

Code for systematically downloading available ARC2 weather .tif files for east Africa and shaping the data for inclusion in the One Acre Fund data warehouse.

One Acre Fund has been using aWhere for weather data access for the simplicity of their GPS based API. However, we're looking to do more with site GPS and weather data and have not seen aWhere to be a superior product in terms of quality. Therefore we are accesing publically available weather datasets so that we can more easily communicate and share with collaborators on weather data related projects.

This code is the example / starter code for accessing a publically available weather dataset and formatting the data for 1AF's use cases. Eventually we'll be adding these data or CHIRPS into the organizational data warehouse so that we can automate extracting long and short term weather averages using site GPS.

How to install

To install this package, please do the following:

1. You'll need to create a personal access token (PAT) on github to download the package from github.

• Go to the tokens page on github to create a token. Be sure to select the repo scope so that it works!
• Then, you'll need to edit the .Renviron file on your computer to make that PAT available to R when it tries to download the package.
• To do this, type usethis::edit_r_environ() in R to edit the .Renviron file.
• In the .Renviron file type GITHUB_PAT = "longstringoflettersandnumbersyougetfromgithub". (What goes in the quotes is the PAT you got from GitHub). Then save and close that file. You might have to restart R for the changes to take effect.

2. Open R script and download the package using devtools::install_github(one-acre-fund/arc2weather). (You can check that the GITHUB_PAT is working as it should by first typing Sys.getenv("GITHUB_PAT") in the R console.)
3. The package should now be available for your use!

How to use

The extract_weather_data function takes several inputs:

• start_date- the lower bound date for which you want weather values.
• end_date - the upper bound date for which you want weather values.
• gps_file - the data file with the GPS points
• lat_col - the latitude column in the GPS data
• lon_col - the longitude colum in the GPS data

You can find example GPS data here: link and the file is named kenya_gps_2019.rds. These are the inputs I've been using to test the package but of course we should try additional GPS to stress test the code! The code currently assumes the gps_file is in the same working directory but future versions will accommodate other file locations.

Running the code looks like:

# first make suer you install the arc2weather package following the instructions above.
library(arc2weather)

weatherValues <- extract_weather_data(start_date = "2010-01-01", 
                                      end_date = "2010-12-31", 
                                      gps_file = gpsData, 
                                      lat_col = "Latitude", 
                                      lon_col = "Longitude")

# one year timing
# user  system elapsed
# 374.072 106.877 510.620

How to find and download data (depreciated)

I'm keeping this here for now but the new API system should mean that we don't have download the data locally first. The files were getting too big to load into R for extraction.

The ARC2 data are available at this FTP address: ftp://ftp.cpc.ncep.noaa.gov/fews/fewsdata/africa/arc2/. The data are most easily avialable in binary format so the code will download the binary format and convert it to raster according to the process outlined in the ARC2 readme (see FTP link for more details).

I've downloaded a large portion of the data to this Google Drive folder. You will need local access to these data if you want to extract weather values from the data meaning you'll need to download the folder to computer.

The function update_arc_weather_data() will look at the available data, determine what data we already have, and then go and pull the latest data. This function assumes that you're pointing it at the Google Drive folder linked above! If you sync that Google drive folder to your computer, it should already have the subfolders you need to update the data and proceed with data extraction. A couple notes:

• The arc2_weather_data folder on Google Drive is a shared data folder for 1AF users. This means that 1AF users shouldn't need to update this data folder themselves since we'll have it regularly updating in this central location. That also implies that any change to this folder affects all users!
• The functions as they currently stand (1.20.19) are specifically set up with the 1AF structure in mind meaning that the download code won't be as useful for outside users. A medium term goal is to update these download functions so that the code can be used by others.

As an example of how to download data:

# this is where the data are on my computer
arc2_weather_directory <- "/Users/mlowes/Google Drive/analyses/weather_analyses/arc2_weather_data"

update_arc_weather_data(arc2_weather_directory)

This will download the latest available weather data, save the data with the others, and update the reference list of the data that we have. This leaves us ready to extract updated values from the data for our calculations!

Todo

• Future stress testing of other GPS formats.
• Add flexible location for gps_file

References

rnoaa - R code to access other NOAA data sets. ARC2 isn't one of them.

CHIRPS data FTP- ftp://ftp.chg.ucsb.edu/pub/org/chg/products/CHIRPS-2.0/ - will expand tools to include this as well

arc2weather's Issues

what to do if the data is already up to date

Consider adding in some sort of check that there's actually new data to access

consider accessing CHIRPS as well

ftp://ftp.chg.ucsb.edu/pub/org/chg/products/CHIRPS-2.0

NAs in gps data

Provide logic to deal with NAs in the GPS data and communicate back to user what is being done.

I think we'll check if there are NAs and if so, remove them based on Lat or Lon variable and then check again. If still NAs, then stop process.

complete.cases1 for two variables?

integrate API and incremental loading

only download and extract from new values

way to avoid storing list of data multiple times

I don't want to store all slices and then them combined data multiple times nor load it and combine it over and over. What's the best way to approach this?

set up batch call for API requests

Organize functions and file to download the data, save it, and weekly download the latest data.