cont-limno / lagosne

Interface to the LAke multi-scaled GeOSpatial & temporal database :earth_americas:

LAGOSNE

The LAGOSNE package provides an R interface to download LAGOS-NE data, store this data locally, and perform a variety of filtering and subsetting operations.

LAGOS-NE contains data for 51,101 lakes and reservoirs larger than 4 ha in 17 lake-rich US states. The database includes 3 data modules for: lake location and physical characteristics for all lakes; ecological context (i.e., the land use, geologic, climatic, and hydrologic setting of lakes) for all lakes; and in situ measurements of lake water quality for a subset of the lakes from the past 3 decades for approximately 2,600-12,000 lakes depending on the variable (see Soranno et al. 2017 below).

Installation

# install stable version from CRAN
install.packages("LAGOSNE")

# install development version from Github
# install devtools if not found - install.packages("devtools")
# devtools::install_github("cont-limno/LAGOSNE", dependencies = TRUE)

Data

The lagosne_get function downloads the LAGOSNE files corresponding to the specified version from the EDI data repository. Files are stored in a temporary directory before being “compiled” to an R data format in the location specified by the dest_folder argument. Recommended setting is lagos_path(). Data only needs to be downloaded one time per version per machine. Each LAGOSNE module has a unique version number. However, only the limno module has been dynamically updated. Therefore the LAGOSNE R package uses the limno module version number to check-out specific datasets. The latest version of the LAGOSNE dataset is 1.087.3.

library(LAGOSNE)
lagosne_get(dest_folder = lagos_path())

Usage

Load Package

library(LAGOSNE)

Load data

The lagosne_load function returns a named list of data.frame objects. Use the names() function to see a list of available data frames names(dt).

dt <- lagosne_load()
names(dt)

#>  [1] "county"               "county.chag"          "county.conn"         
#>  [4] "county.lulc"          "edu"                  "edu.chag"            
#>  [7] "edu.conn"             "edu.lulc"             "hu4"                 
#> [10] "hu4.chag"             "hu4.conn"             "hu4.lulc"            
#> [13] "hu8"                  "hu8.chag"             "hu8.conn"            
#> [16] "hu8.lulc"             "hu12"                 "hu12.chag"           
#> [19] "hu12.conn"            "hu12.lulc"            "iws"                 
#> [22] "iws.conn"             "iws.lulc"             "state"               
#> [25] "state.chag"           "state.conn"           "state.lulc"          
#> [28] "buffer100m"           "buffer100m.lulc"      "buffer500m"          
#> [31] "buffer500m.conn"      "buffer500m.lulc"      "lakes.geo"           
#> [34] "epi_nutr"             "lakes_limno"          "lagos_source_program"
#> [37] "locus"

Locate tables containing a variable

query_lagos_names("secchi")

#> [1] "epi_nutr"

Preview a table

head(dt$state)
#>   state    state_name state_zoneid state_lat state_long state_pct_in_nwi
#> 1    IA          Iowa     State_13  42.07456  -93.49983              100
#> 2    MA Massachusetts      State_2  42.25762  -71.81240              100
#>   state_ha_in_nwi state_ha
#> 1        14573561 14573561
#> 2         2101262  2101262

Preview a specific lake

lake_info(name = "Pine Lake", state = "Iowa")
# or using a lagoslakeid
# lake_info(lagoslakeid = 4389)

#>   lagoslakeid     nhdid  nhd_lat  nhd_long      lagosname1 meandepth
#> 1        4510 155845265 42.37833 -93.05967 UPPER PINE LAKE      2.21
#>   meandepthsource maxdepth maxdepthsource legacyid gnis_name lake_area_ha
#> 1    IA_CHEMISTRY     4.88   IA_CHEMISTRY      122 Pine Lake     36.07355
#>   lake_perim_meters nhd_fcode nhd_ftype iws_zoneid hu4_zoneid hu6_zoneid
#> 1          5671.001     39004       390  IWS_51040     HU4_57     HU6_78
#>   hu8_zoneid hu12_zoneid edu_zoneid county_zoneid state_zoneid elevation_m
#> 1    HU8_400   HU12_3008     EDU_23    County_275     State_13      300.23
#>   state state_name state_lat state_long state_pct_in_nwi state_ha_in_nwi
#> 1    IA       Iowa  42.07456  -93.49983              100        14573561
#>   state_ha lakeconnection   iws_ha
#> 1 14573561      DR_Stream 3593.379

Read table metadata

help.search("datasets", package = "LAGOSNE")

Package	Topic	Title
LAGOSNE	chag	Climate, Hydrology, Atmospheric, and Geologic (CHAG) Datasets
LAGOSNE	classifications	LAGOSNE Spatial Classifications Metadata
LAGOSNE	conn	Connectivity Datasets
LAGOSNE	epi_nutr	Epilimnion Water Quality Data
LAGOSNE	lagos_source_program	LAGOSNE sources
LAGOSNE	lagoslakes	Lake Geospatial Metadata
LAGOSNE	lakes_limno	Metadata for Lakes with Water Quality
LAGOSNE	locus	Metadata for all lakes > 1ha
LAGOSNE	lulc	Land Use Land Cover (LULC) Data Frames

Select data

lagosne_select is a convenience function whose primary purpose is to provide users with the ability to select subsets of LAGOS tables that correspond to specific keywords (see LAGOSNE:::keyword_partial_key() and LAGOSNE:::keyword_full_key()). See here for a comprehensive tutorial on generic data.frame subsetting.

# specific variables
head(lagosne_select(table = "epi_nutr", vars = c("tp", "tn"), dt = dt))
#>       tp     tn
#> 1  29.00     NA
#> 2 136.56 3521.7
head(lagosne_select(table = "iws.lulc", vars = c("iws_nlcd2011_pct_95"), dt = dt))
#>   iws_nlcd2011_pct_95
#> 1                0.04

# categories
head(lagosne_select(table = "locus", categories = "id", dt = dt))
#>   lagoslakeid iws_zoneid hu4_zoneid hu6_zoneid hu8_zoneid hu12_zoneid
#> 1        3201       <NA>     HU4_11     HU6_12     HU8_47  HU12_16312
#> 2        4510  IWS_51040     HU4_57     HU6_78    HU8_400   HU12_3008
#>   edu_zoneid county_zoneid state_zoneid
#> 1     EDU_27    County_331      State_2
#> 2     EDU_23    County_275     State_13
head(lagosne_select(table = "epi_nutr", categories = "waterquality", dt = dt))
#>    chla colora colort dkn doc nh4 no2 no2no3 srp tdn tdp tkn     tn toc ton
#> 1 16.60     60     NA  NA  NA  NA  NA     NA  NA  NA  NA  NA     NA  NA  NA
#> 2 30.64     NA     NA  NA  NA  NA  NA 1619.6  NA  NA  NA  NA 3521.7  NA  NA
#>       tp secchi
#> 1  29.00   1.70
#> 2 136.56   0.65
head(lagosne_select(table = "hu4.chag", categories = "deposition", dt = dt)[,1:4])
#>   hu4_dep_no3_1985_min hu4_dep_no3_1985_max hu4_dep_no3_1985_mean
#> 1               7.2171              10.0448                7.9366
#> 2               9.5538              21.1791               15.5290
#>   hu4_dep_no3_1985_std
#> 1               0.3868
#> 2               2.2330

# mix of specific variables and categories
head(lagosne_select(table = "epi_nutr", vars = "programname", 
                    categories = c("id", "waterquality"), dt = dt))
#>   programname lagoslakeid  chla colora colort dkn doc nh4 no2 no2no3 srp tdn
#> 1      MA_DEP        3201 16.60     60     NA  NA  NA  NA  NA     NA  NA  NA
#> 2     IA_CHEM        4510 30.64     NA     NA  NA  NA  NA  NA 1619.6  NA  NA
#>   tdp tkn     tn toc ton     tp secchi eventida10873
#> 1  NA  NA     NA  NA  NA  29.00   1.70         45773
#> 2  NA  NA 3521.7  NA  NA 136.56   0.65         64904

Published LAGOSNE subsets

# Oliver et al. 2015
lagos_get_oliver_2015()
head(lagos_load_oliver_2015())

# Collins et al. 2017
lagos_get_collins_2017()
head(lagos_load_collins_2017())

Legacy Versions

R Package

To install versions of LAGOSNE compatible with older versions of LAGOS-NE run the following command where ref is set to the desired version (in the example, it is version 1.087.1):

# install devtools if not found
# install.packages("devtools")
devtools::install_github("cont-limno/LAGOSNE", ref = "v1.087.1")

References

Oliver, SK, PA Soranno, CE Fergus, T Wagner, K Webster, CE Scott, LA Winslow, J Downing, and EH Stanley. 2015. “LAGOS - Predicted and Observed Maximum Depth Values for Lakes in a 17-State Region of the U.S.” https://dx.doi.org/10.6073/pasta/edc06bbae6db80e801b6e52253f2ea09.

Soranno, P.A., Bacon, L.C., Beauchene, M., Bednar, K.E., Bissell, E.G., Boudreau, C.K., Boyer, M.G., Bremigan, M.T., Carpenter, S.R., Carr, J.W. Cheruvelil, K.S., and … , 2017. LAGOS-NE: A multi-scaled geospatial and temporal database of lake ecological context and water quality for thousands of US lakes. GigaScience, https://doi.org/10.1093/gigascience/gix101

Soranno, PA, EG Bissell, KS Cheruvelil, ST Christel, SM Collins, CE Fergus, CT Filstrup, et al. 2015. “Building a Multi-Scaled Geospatial Temporal Ecology Database from Disparate Data Sources: Fostering Open Science and Data Reuse.” Gigascience 4 (1). https://dx.doi.org/10.1186/s13742-015-0067-4.

Stachelek J., Oliver S. 2017. LAGOSNE: Interface to the Lake Multi-scaled Geospatial and Temporal Database. R package version 1.1.0. https://cran.r-project.org/package=LAGOSNE

Soranno P, Cheruvelil K. 2017. LAGOS-NE-LOCUS v1.01: a module for LAGOS-NE, a multi-scaled geospatial and temporal database of lake ecological context and water quality for thousands of U.S. Lakes: 1925–2013. Environmental Data Initiative. https://doi.org/10.6073/PASTA/0C23A789232AB4F92107E26F70A7D8EF

Soranno P, Cheruvelil K. 2019. LAGOS-NE-LIMNO v1.087.3: a module for LAGOS-NE, a multi-scaled geospatial and temporal database of lake ecological context and water quality for thousands of U.S. Lakes: 1925–2013. Environmental Data Initiative. https://doi.org/10.6073/PASTA/08C6F9311929F4874B01BCC64EB3B2D7.

Soranno P, Cheruvelil K. 2017. LAGOS-NE-GEO v1.05: a module for LAGOS-NE, a multi-scaled geospatial and temporal database of lake ecological context and water quality for thousands of U.S. Lakes: 1925–2013. Environmental Data Initiative. https://doi.org/10.6073/PASTA/16F4BDAA9607C845C0B261A580730A7A

lagosne's Issues

Allow user to select groups of columns

In addition to the ability select columns by name, allow the user to select pre-defined groups of columns within each table. For example, "atmospheric deposition" that may include multiple variables.

Vignette building breaks CI (and CRAN) checks

The vignettes as currently written require that the full LAGOS dataset is installed and available to lagos_load. I think this is good because it allows us to describe the contents of the data product. However, automated build testing via CI services (and eventually CRAN) breaks because the they don't (and probably should not) have the data available. The only solution I can come up with is to make the vignettes static. That is to pre-build all the vignette figures and tables and display the code chunks without running them (eval = FALSE).

Create function(s) to download remote raw tables and "compiled" R objects

Store data using https://github.com/hadley/rappdirs

Possibly implement some of the ideas in https://github.com/richfitz/datastorr/blob/master/vignettes/datastorr.Rmd

Make lagos_compile and lagos_load faster by creating more than one rds file

Should this be done at the table level or the module level?

Standardize documentation

Some functions have a title in the documentation and other do not. Consider following the rules at: http://style.tidyverse.org/code-documentation.html

Create a function that generates a subset of LAGOS for `testthat`

Something like the first 10 lines of every table

Add a function to turn a table into a spatial object

LAGOS does not fail well

See #22, and the output of devtools::test(). Need to throw helpful error messages when queries try to return data that does not exist.

lagos_select locks up if provided bad inputs for vars

Need to sanitize inputs

lagos_select does not allow for selecting only from limno or only from geo

Refactoring the code will need to be done carefully.

add descriptions of hu8 etc to `lagos_select` man page under scale

Create a `citation` function/file

See http://r-pkgs.had.co.nz/inst.html#inst-citation

Include functions to download and query previously published LAGOS subsets?

https://portal.lternet.edu/nis/mapbrowse?packageid=knb-lter-ntl.320.4

http://datadryad.org/resource/doi:10.5061/dryad.75s9s

Add a lagos_select keyword for sample info

sampling event, lagoslakeid, etc.

All of the tables returned by lagos_load do not have corresponding help files

For example, ?locus does not yield any results.

Also buffer100m.lulc buffer500m.lulc lakes.geo lagos_source_program

Add a function to return table names given match between search string and column headers

HUC Zoneids are loaded as factors not character strings

This was done to be consistent with the legacy loading scripts but can be a real pain for analysis. Is there any reason not to load as character strings instead?

List lagos_select keywords in documentation?

The keywords in LAGOS:::keyword_partial_key() are not included in any of the user-facing documentation.

Include detection limit handling as an argument to `lagos_select`?

Create function(s) to load and type-check the individual txt files for LAGOS-limno and LAGOS-geo

This is the bulk of the old LAGOS R package but it could be much cleaner by using functions to reduce redundancy versus copy-paste.

Update docs to reflect upgrade to 1.087

Create vignettes to document common filtering/subsetting operations

See http://r-pkgs.had.co.nz/vignettes.html

Identify existing multitable Gigascience datasets of a similar size to test remote downloading

`lagos_select` doesn't work with scale = "iws"

Refactor lagos_select as a keyword/lakeid* helper only?

Provide user with all column names within each table + metadata

@jsta where is the best place for this information? As a user, I could imagine wanting this to be in two places: 1) some sort of documentation listing each variable within each table, along with some metadata (units, plain English description, etc). We could have documentation for each table, but where (in the package structure) would this go? 2) in a table format, similar to the info table that is currently imported with the rds file.

Coordinatize uses the sp package instead of the sf package

Add interactive select functionality?

This was present in the legacy code base.

lagos_compile fails

LAGOS:::lagos_compile(version = "1.054.1", format = "rds")

fails with error message:
Error in gzfile(file, mode) : cannot open the connection
In addition: Warning message:
In gzfile(file, mode) :
cannot open compressed file 'C:\Users\Samantha\AppData\Local\LAGOS\LAGOS/data_1.054.1.rds', probable reason 'No such file or directory'

R version 3.3.1 (2016-06-21)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 7 x64 (build 7601) Service Pack 1

sessionInfo()
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] LAGOS_1.054.1

loaded via a namespace (and not attached):
[1] lazyeval_0.2.0 magrittr_1.5 R6_2.1.2 assertthat_0.1 rsconnect_0.4.3
[6] DBI_0.5-12 tools_3.3.1 dplyr_0.5.0 rappdirs_0.3.1 tibble_1.1
[11] Rcpp_0.12.7

rappdirs::user_data_dir("LAGOS")

"C:\Users\Samantha\AppData\Local\LAGOS\LAGOS"

keywords should pull additional info like censor codes and lagos flags

exactly which ones is an open question

Avoid re-loading existing data in lagos_load

Add checks to see if data are already loaded. This is what datastorr/storr does well.

Provide users with metadata about LAGOS

One idea might be to use the units package

Update docs to reflect upgrade to 1.054.2

Some tables have underscore separators now

epi.nutr -> epi_nutr
lakes.limno -> lakes_limno

Return dates as formatted character strings rather than factors

Add ability to compile from an arbitrary folder location

Assign units to fields with the units package

exports converted from txt to csv

the data compilation functions will need to be updated

add variable aliases?

We might consider adding additional alias terms relative to the datasets. For example, we might list "chla", "colora", and "doc" as aliases for the epi.nutr table. This would enable ??LAGOS::chla

Provide users with warnings about censor codes

Change repository description to match package DESCRIPTION

LAke multi-scaled GeOSpatial & temporal database -> Tools for Interacting with the Lake Multi-scaled Geospatial and Temporal Database

• @jiayuzhou can you do this? I cannot because I do not have administrator level control of the repository.

buffer table headers not recognized

Tables "lakes4ha.buffer500m" and "lakes4ha.buffer100m" do not import with column names (column names in row 1).

Pull all columns from table if no columns or categories specified

See #15

Refactor preprocessing(merging/selecting) functions

At the very least, make preprocessing functions operate on column names and not column numbers

A lot of this code may have reinvented the wheel. It is likely that we can simplify a lot of this by wrapping the dplyr package.

Question: Are the generated lake watershed polygons available through the LAGSO package?

Inquiring minds want to know.

column names for buffer metadata tables imported incorrectly

In tables lakes4ha_buffer100m and lakes4ha_buffer500m, column is "lagoslakeid" after importing through package, but column name in geo v1.040 is "lakes4ha_buffer100m_lagoslakeid" and "lakes4ha_buffer100m_lagoslakeid".

Add progress notifications to `lagos_get` and `lagos_compile`

allow users to input multiple 'scale' options in lagos_select

make lagos_get_oliver_2015 more broadly applicable to other published LAGOS datasets

A nice feature of the package might be to compile ALL previously published LAGOS data (I think all of it is on the LTER portal). So the function could be something like lagos_published(paper = c("Oliver2015", "Lottig 2014"))

Access to package

@jsta -- I just tried to get Emily Stanley (@ehstanley) set up with the R package and she doesn't have access. How do I change this?