inbo / iassetr

Interact with the iAsset api using R

iassetR

The goal of iassetR is to interact with the iAsset api using R. Currently only reading from the iAsset API is possible. But we intend to expand the scope to writing data as well if this is somewhat securely possible.

This package is developed for use in case of VespaWatch, which uses iAsset under the umbrella of the RIPARIAS project. Early in the development a wider use, beyond the even the limits of RIPARIAS, was envissioned resulting in a name change from vespawatchR to iassetR.

The current version of the package should be usable by all users of iAsset however keep in mind it has only been tested for vespawatch use. When you encounter issues please raise an issue.

Installation

You can install the development version of iassetR from GitHub with:

# install.packages("devtools")
devtools::install_github("inbo/iassetR")

Example

This is a basic example which shows you how to solve a common problem:

library(iassetR)
## basic example code

iassetr's Issues

write tests for all functions

We probably want to cache, anonymised responses, we could use vcr for this, similar to etn and rgbif

https://github.com/vcr/vcr

Not an urgent issue

write a readme

Ideally we'd also add a vignette on how to use the package, but this is something for beyond v0.1.

Things to mention in the readme:

• Why did we make this package, mention that we want it to be broadly applicable for all iAsset users, but that our focus is currently on Vespa-Watch and RIPARIAS
• What is the scope of the package (currently reading records, not changing them)
• Refer to the iAsset API documentation
• What can you do with the package; functions; get_access_token() for a token, get_records() to get records and get_fields() to get information about the fields
• (optional) Mention Vespa-Watch as funding?
• An example (excluding passwords, usernames or keys!) or how to get records out of iAsset

Come up with a better name for this package

This package is more general than just vespa-watch, it'll work just as well for other domains on iAsset. And in fact, we are planning to use it for our other project on our domain as well.

I'm open to any suggestions!

attatch field names from get_fields() to output of get_records()

I currently fetch the field names, but we want to attach these to the output of the tibble

Write a vignette to demonstrate the usage of the package

We aught to demonstrate how to use the functions in this package, in a logical order. So users who aren't as familiar with iAsset or APIs in general can use it right away.

Create citation.cff

Not very difficult with cffr::cff_write() but I'd prefer a solution that will automatically update the citation.cff with every release

Better column classes for output of `get_records()`

Currently the tibble that get_records() returns has character columns for everything except list columns, we have column type information from get_fields(), we could translate these into R equivalents and do some parsing/mapping to make the output a lot more user friendly.

For example: toestemming_gegevensgebruik == "on" might make more sense as toestemming_gegevensgebruik == TRUE, we could use date and time fields for datum_tijd_registratie and datum_observatie to save users on having to to these conversions themselves.

cleanup hardcoded inputs

Currently some functions contain hardcoded inputs these should be removed

Implement filtering in `get_records()`

Currently get_records() retreives all records for a customInspection, it would be nice if we could also use the custom filters we have setup in the web interface, such as for example: 2. Valideren

The API docs: https://api.iasset.nl/documentation#jump-Inspections_2FCustomInspection-getCustomInspections, mention that this is possible using a different request first:

Suggest using “GetSchouwObjects” request first to get the certain objects for which You want to see the custom inspections.

httr2 doesn't seem to work for get_records(), but httr does

I'm used to writing my httr2 requests like this:

get_records <- function(inspection_name = "Vespa-Watch",
                        access_token = get_access_token(quiet = TRUE)) {
  # check input params
  assertthat::assert_that(assertthat::is.string(access_token))
  assertthat::assert_that(assertthat::is.string(inspection_name))
  # get the inspection_id for the custom inspection
  inspection_fields <- get_fields(access_token = access_token,
                                  name = inspection_name)
  # build a request and perform it
  get_records_request <-
    httr2::request("https://api.iasset.nl/getCustomInspections") %>%
    httr2::req_body_form(
      access_token = access_token,
      "inspection_ids[0]" = inspection_fields$id,
      version = "9.7"
    )
}

but it doesn't work, while this in httr does:

body = list(
  'access_token' = access_token,
  'version' = '9.7',
  'inspection_ids[0]' = inspection_fields$id
)

res <-
  httr::VERB(
    "POST",
    url = "https://api.iasset.nl/getCustomInspections",
    body = body,
    encode = 'multipart'
  )

# cat(content(res, 'text'))

res %>% httr::content()

• is this a bug in httr2 ?
• Is my httr2 code wrong?

Add parameter for domain in `get_access_token()`

Currently the package only supports the riparias domain, and while this can be the default, I think domain should be able to change so the package can be used by other iAsset users.

Update contributors

Add Sander to contributors

What do the values for `toestemming_gegevensgebruik` mean, and should we recode them?

They are currently, 1,on or empty:

toestemming_gegevensgebruik	n
1	87
on	9432
NA	1

To be investigated

What is the unit for `tijd_ter_plaatse`

Currently we return this as an integer, encoded as a character. There are only a few values in use, what do they mean?

tijd_ter_plaatse	n
10	4
120	1
1215	1
15	8
20	5
30	3
40	1
45	1
5	3
60	3
90	1
NA	9489

It would probably be more user friendly to return these values either with a unit, or as a lubridate::duration()

Get Keyring to work with github actions

I'am experiencing issues with getting get_records() to run with github actions. I get the following error:

Error in b__file_keyring_autocreate(self, private, keyring) : 
  The 'system' keyring does not exists, create it first!
Calls: source ... b_file_list -> <Anonymous> -> b__file_keyring_autocreate
Execution halted

in https://github.com/inbo/aspbo/actions/workflows/get_vespa_velutina_management.yaml. My latest run: https://github.com/inbo/aspbo/actions/runs/8615691797/job/23613099659

its seems the error occurs when the workflow runs the following code:
https://github.com/inbo/aspbo/blob/afee4e3527ef617f9a1c6074242b838989d9ed71/src/Vespa%20velutina%20management/get_data_from_iAsset.Rmd#L33-L39

When I look at the keyring readme I see the following remark:

The envvar backend doesn’t support custom keyrings, so if you’re using one locally you’ll need to use the default keyring on GitHub.

This makes me suspect adding service = "iasset_password" is the cause of this issue.

PS I added "iasset_username" & "iasset_password" as github Actions secrets on the repository level.

Add progress reporting to `get_records()`

get_records() takes a while to run, this is because it currently does two requests including a big one to actually fetch all the records in one go. With filtering #16 we'll need another request.

• We could add a verbose = TRUE argument to keep the user up to date on what the function is doing
• Ideally we'd want some sort of progress bar on the slowest request, I expect this request will get a lot slower/bigger next year. And it's 10MB already.

get_records(): handle list columns

Currently we have a few multivalue in the API output, we still need to handle these more elegantly before we can output to a tibble.

Currently we have explored tibble::enframe(), and tidyr::pivot_wider()

res %>% httr::content() %>% 
  purrr::pluck("returndata") %>% 
  head(5) %>% 
  purrr::map(~purrr::pluck(.x, "data")) %>%
  # purrr::map(~purrr::map_dfc(.x, function(element){element}))
  purrr::map(~tibble::enframe(.x)) %>% 
  purrr::map_dfr(~ tidyr::pivot_wider(.x, names_from = name))

But I personally think it should probably be handled with a combination of purrr::map_dfc and purrr::map_dfr

Optionally return verbatim column names from iAsset

Currently field names are sanitized by way of janitor::clean_names(), I think for some users it'll be nice to turn this behaviour off via a function argument. something like clean_names = TRUE by default.

transfer maintainer role to sander

`get_record()` doesn't give an ID for each observation.

Currently, there is no unique ID for each observation.

I feel like the object_id, inspection_id and insp_order are the best options to keep in the final data as they are used in the editCustomInspection which seams to be the API call used to edit observations. More over, the insp_order are the IDs currently used in VespaR.

Those three fields are removed by this step :
https://github.com/inbo/iassetR/blob/818e923e16b7ba54f96b7e7334b7475e53bcb148/R/get_records.R#L41C5-L42C46

I'm planning on doing a PR if that's a feature you agree to have.

pass credentials via sys env

1 question though I'm looking into automating retrieval of vespawatch nests using github actions and I was just wondering if this will work if we cache the token instead of being able to provide a password as environment variable.

This is an important design decision to keep in mind, can you make an issue for further discussion?

If we retrieve personal information via the API, we might need to make sure no data is stored unencrypted on the runner or in artefacts, and we might be obliged to process data in the EU only. I'm not sure on the specifics but we need to thread lightly. We are capable of using our own custom runners via AWS in a EU region, but it'll take a bit of doing and have a slight cost.

Originally posted by @PietrH in #6 (comment)

Better error message for wrong username

The failure state when you enter the wrong username isn't very elegant:

get_records(access_token = get_access_token(username = "[email protected]"))

#>Error: access_token is not a string (a length one character vector).

add diego to contributors

his orcid: https://orcid.org/0009-0003-2454-8521

cache the access token in the system environement

Maybe store the access token for a bit, similar to etn and rgbif

Convert data from answercode to answer

Currently fields of fieldtype == "select" or "radio" (maybe also others) contain a code related to their answer in the output of get_records. It would be usefull to have the actual answers instead.
The actual answers can be found in the "options" column of get_fields "fields" slot.
There is no data column present to link the answer with the answer code.

Pay attention: the content of the column "fieldlabel" is not snakecase while the column names from get_records are!
Pay attention: some select/radio columns from get_records are of class "list" these merit a different approach!

Some testing from myself:
https://github.com/inbo/aspbo/blob/ae995deb42686abc644d6e02f290f78c4204d1d8/src/get_data_from_iAsset.Rmd#L27-L49