Develop function to extract IWR tidal creek data from FDEP FTP about tbeptools

[mikewessel]

Use creek_pop.sas7bdat file WBID field in tbeptools as list or in one to many join to subset data based on WBIDs within SW FL tidal creeks population

[fawda123]

@mikewessel is this the only step to distill the entire run 56 database to iwr56_tidalcreeks.sas7bdat?

[mikewessel]

I believe so. I have been going through the tbeptools package and don’t see when the region is assigned within iwrraw so it must be assigned by a join? I have a few questions, perhaps we can do a call this week to discuss a few things?

[fawda123]

@mikewessel partially there, but running into memory issues on import...

# specify run
run <- 'run60'

# check if url exists
url <- paste0('http://publicfiles.dep.state.fl.us/dear/kristin/', run, '/')
urlexst <- RCurl::url.exists(url)
if(!urlext)
  stop('URL for ', url, ' doesnt exist')

# get file list, probably overkill because the file name doesn't change
fls <- RCurl::getURL(url, verbose = T, ftp.use.epsv = F, dirlistonly = T, crlf = T)
zip <- gsub('^.*\\">(.*\\.zip)<.*$', '\\1', fls)
zip <- paste0(url, zip)

# download and unzip to temp directory
tmp1 <- tempfile()
tmp2 <- tempfile()
download.file(zip, destfile = tmp1)
unzip(tmp1, exdir = tmp2)

# attempt to read sas file
dat <- haven::read_sas(list.files(tmp2, full.names = T))
### Error: cannot allocate vector of size 318.8 Mb

[mikewessel]

Don’t think the link below gets you exactly where you want to go but thinking using sql or python might be a workaround? I’ll investigate. https://tech.lanesnotes.com/2012/06/importing-sas7bdat-file-into-sql-server.html?m=1

…

Sent from my iPhone

On Jan 25, 2021, at 6:24 PM, Marcus W Beck ***@***.***> wrote:  @mikewessel partially there, but running into memory issues on import... # specify run run <- 'run60' # check if url exists url <- paste0('http://publicfiles.dep.state.fl.us/dear/kristin/', run, '/') urlexst <- RCurl::url.exists(url) if(!urlext) stop('URL for ', url, ' doesnt exist') # get file list, probably overkill because the file name doesn't change fls <- RCurl::getURL(url, verbose = T, ftp.use.epsv = F, dirlistonly = T, crlf = T) zip <- gsub('^.*\\">(.*\\.zip)<.*$', '\\1', fls) zip <- paste0(url, zip) # download and unzip to temp directory tmp1 <- tempfile() tmp2 <- tempfile() download.file(zip, destfile = tmp1) unzip(tmp1, exdir = tmp2) # attempt to read sas file dat <- haven::read_sas(list.files(tmp2, full.names = T)) ### Error: cannot allocate vector of size 318.8 Mb — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[fawda123]

Spent most of the morning trying to get this working, but no luck yet: https://github.com/bnosac/spark.sas7bdat

Even if it does work, it would probably take some effort to make it portable (i.e., automate on different machines). I still think it's worth getting setup though...

[mikewessel]

The easiest route is to get DEP to export their run as a txt file. I can provide code for them and in reality it's just uploading two files each time instead of one. Alternatively I can export the sas dataset as a txt file each time for now. If you want to try that let me know. Not sure what the options are for working with the MS ACCESS db they provide but that could be an alternative as well if hurdles are lower. This is probably a stretch but what about tapping into EPA mainframe and using their SAS licence to call sas code from R, convert to sas file to txt file and then import that way? I'm sure that wouldn't fly, they'd think you were Cozy Bear ....

[fawda123]

@mikewessel do you think it's feasible for them to upload a text file? Wouldn't that be much larger than the SAS file?

I'd rather not go the Access route. I've done it but it makes the workflow so much more vulnerable.

[mikewessel]

I exported run 60 as a txt file. It is 1/3 smaller (6gb). I tried to read in with readr package but got a weird error boost::interprocess_exception. No idea what that means. I am uploading file to our ftp site and will send u a link/pass to tbep.org account. Worth a shot?

…

Sent from my iPhone

On Jan 29, 2021, at 8:03 AM, Marcus W Beck ***@***.***> wrote:  @mikewessel do you think it's feasible for them to upload a text file? Wouldn't that be much larger than the SAS file? I'd rather not go the Access route. I've done it but it makes the workflow so much more vulnerable. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[mikewessel]

Exporting from sas to binary also may be possible

…

Sent from my iPhone

On Jan 29, 2021, at 12:14 PM, ***@***.*** wrote: I exported run 60 as a txt file. It is 1/3 smaller (6gb). I tried to read in with readr package but got a weird error boost::interprocess_exception. No idea what that means. I am uploading file to our ftp site and will send u a link/pass to tbep.org account. Worth a shot? Sent from my iPhone >> On Jan 29, 2021, at 8:03 AM, Marcus W Beck ***@***.***> wrote: >> >  > @mikewessel do you think it's feasible for them to upload a text file? Wouldn't that be much larger than the SAS file? > > I'd rather not go the Access route. I've done it but it makes the workflow so much more vulnerable. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub, or unsubscribe.

[fawda123]

@mikewessel I think the data.table package is the way to go for large text files. I can get it manageable using this code. Is the text file something that DEP can host?

library(data.table)
library(tbeptools)

tmp <- fread('~/Desktop/iwr2002_pv.txt', sep = '\t')
tmp2 <- tmp[wbid %in% tidalcreeks$wbid, c('wbid', 'class', 'sta', 'year', 'month', 'day', 'time', 'depth', 'depth2', 'param', 'masterCode', 'result')]

rm(list = 'tmp')

save(tmp2, file = 'data/tmp2.RData', compress = 'xz')

[fawda123]

This may be possible to query through Water Atlas:

From Adok:

We currently have the Impaired Waters data as a part of the Water Bodies WaterQuality API. It is currently bundled and query-able by Waterbody id within the water atlas.

Here is the API: https://api.wateratlas.usf.edu/sandbox/index#!/Waterbodies/WaterBodyDetails_GetWaterQualityData
Example for Lake Thonotosassa https://api.wateratlas.usf.edu/waterbodies/5383/WaterQuality?s=2

I'll take a stab, sometime in the next couple of weeks, at making this API be a little more user friendly (Query by WBID/county & return a little more detail such as Run numbers etc.)

[mikewessel]

For DEP to generate the iwr run as a txt file they would need to insert a single line of code (below) and then upload the resultant txt file. proc export data=C.IWR2002_PV outfile="IWR2002_PV.txt" dbms=tab replace; run; This is more in line with how we generated the dataset for the creeks work. The wateratlas may also work just fine but does cross several geographical boundaries and pay for service agreements and also in some cases houses other dataset that could get duplicated into the database.

…

On Sun, Jan 31, 2021 at 9:53 PM Marcus W Beck ***@***.***> wrote: @mikewessel <https://github.com/mikewessel> I think the data.table package is the way to go for large text files. I can get it manageable using this code. Is the text file something that DEP can host? library(data.table) library(tbeptools) tmp <- fread('~/Desktop/iwr2002_pv.txt', sep = '\t') tmp2 <- tmp[wbid %in% tidalcreeks$wbid, c('wbid', 'class', 'sta', 'year', 'month', 'day', 'time', 'depth', 'depth2', 'param', 'masterCode', 'result')] rm(list = 'tmp') save(tmp2, file = 'data/tmp2.RData', compress = 'xz') — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#37 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADYTBYHRWFYWL7WZXHTVDS3S4YJT3ANCNFSM4VYAGSRQ> .

[fawda123]

Revisiting this again, I think we're almost there: https://tbep-tech.github.io/tidalcreek-stats/Creek_select_tbeptools

Some final questions:

• Is there a corresponding wbid and station file that is released with each IWR run? Run 62 is out now but I can't find a WBID 62 polygon shapefile. Same for stations, but these can be pulled from the .accdb files.
• What is the importance of the newComment column? I couldn't find this in the imported data using the new workflow.
• Are the files TidalCreek_ALL_Line.shp and TidalCreek_ALL_Line_Buff200m permanent? Guessing yes... if so, I will need to include the former in tbeptools, latter can be recreated with st_buffer w/ flat ends.
• Long-term, how do we want to compare result between runs or as a time series? Ten-year moving windows?

[mikewessel]

Working on finding shapefile. Other issues probably easier to discuss at 11.

[fawda123]

Adding this here for the record, IWR spatial data are available here: https://geodata.dep.state.fl.us/, geojson download here: https://geodata.dep.state.fl.us/datasets/FDEP::impaired-waters-rule-iwr-stations/about

[fawda123]

Finally closing this, too tedious to include a function in tbeptools for this. However, this external workfow seems to work well:
https://tbep-tech.github.io/tidalcreek-stats/Creek_select_tbeptools (updated for run 64 in this commit tbep-tech/tidalcreek-stats@6aae2fc).