inbo / iassetr Goto Github PK
View Code? Open in Web Editor NEWInteract with the iAsset api using R
Home Page: https://inbo.github.io/iassetR/
License: Other
Interact with the iAsset api using R
Home Page: https://inbo.github.io/iassetR/
License: Other
This package is more general than just vespa-watch, it'll work just as well for other domains on iAsset. And in fact, we are planning to use it for our other project on our domain as well.
I'm open to any suggestions!
1 question though I'm looking into automating retrieval of vespawatch nests using github actions and I was just wondering if this will work if we cache the token instead of being able to provide a password as environment variable.
This is an important design decision to keep in mind, can you make an issue for further discussion?
If we retrieve personal information via the API, we might need to make sure no data is stored unencrypted on the runner or in artefacts, and we might be obliged to process data in the EU only. I'm not sure on the specifics but we need to thread lightly. We are capable of using our own custom runners via AWS in a EU region, but it'll take a bit of doing and have a slight cost.
Originally posted by @PietrH in #6 (comment)
Currently some functions contain hardcoded inputs these should be removed
Currently the tibble that get_records()
returns has character columns for everything except list columns, we have column type information from get_fields()
, we could translate these into R equivalents and do some parsing/mapping to make the output a lot more user friendly.
For example: toestemming_gegevensgebruik == "on" might make more sense as toestemming_gegevensgebruik == TRUE, we could use date and time fields for datum_tijd_registratie and datum_observatie to save users on having to to these conversions themselves.
Currently fields of fieldtype == "select" or "radio"
(maybe also others) contain a code related to their answer in the output of get_records
. It would be usefull to have the actual answers instead.
The actual answers can be found in the "options" column of get_fields
"fields" slot.
There is no data column present to link the answer with the answer code.
Pay attention: the content of the column "fieldlabel" is not snakecase while the column names from get_records
are!
Pay attention: some select/radio columns from get_records
are of class "list" these merit a different approach!
Some testing from myself:
https://github.com/inbo/aspbo/blob/ae995deb42686abc644d6e02f290f78c4204d1d8/src/get_data_from_iAsset.Rmd#L27-L49
Currently the package only supports the riparias
domain, and while this can be the default, I think domain
should be able to change so the package can be used by other iAsset users.
Add Sander to contributors
Currently field names are sanitized by way of janitor::clean_names(), I think for some users it'll be nice to turn this behaviour off via a function argument. something like clean_names = TRUE
by default.
I currently fetch the field names, but we want to attach these to the output of the tibble
We aught to demonstrate how to use the functions in this package, in a logical order. So users who aren't as familiar with iAsset or APIs in general can use it right away.
Maybe store the access token for a bit, similar to etn and rgbif
We probably want to cache, anonymised responses, we could use vcr for this, similar to etn and rgbif
Not an urgent issue
Currently get_records()
retreives all records for a customInspection, it would be nice if we could also use the custom filters we have setup in the web interface, such as for example: 2. Valideren
The API docs: https://api.iasset.nl/documentation#jump-Inspections_2FCustomInspection-getCustomInspections, mention that this is possible using a different request first:
Suggest using “GetSchouwObjects” request first to get the certain objects for which You want to see the custom inspections.
They are currently, 1
,on
or empty:
toestemming_gegevensgebruik | n |
---|---|
1 | 87 |
on | 9432 |
NA | 1 |
To be investigated
Currently, there is no unique ID for each observation.
I feel like the object_id
, inspection_id
and insp_order
are the best options to keep in the final data as they are used in the editCustomInspection which seams to be the API call used to edit observations. More over, the insp_order
are the IDs currently used in VespaR.
Those three fields are removed by this step :
https://github.com/inbo/iassetR/blob/818e923e16b7ba54f96b7e7334b7475e53bcb148/R/get_records.R#L41C5-L42C46
I'm planning on doing a PR if that's a feature you agree to have.
I'm used to writing my httr2 requests like this:
get_records <- function(inspection_name = "Vespa-Watch",
access_token = get_access_token(quiet = TRUE)) {
# check input params
assertthat::assert_that(assertthat::is.string(access_token))
assertthat::assert_that(assertthat::is.string(inspection_name))
# get the inspection_id for the custom inspection
inspection_fields <- get_fields(access_token = access_token,
name = inspection_name)
# build a request and perform it
get_records_request <-
httr2::request("https://api.iasset.nl/getCustomInspections") %>%
httr2::req_body_form(
access_token = access_token,
"inspection_ids[0]" = inspection_fields$id,
version = "9.7"
)
}
but it doesn't work, while this in httr does:
body = list(
'access_token' = access_token,
'version' = '9.7',
'inspection_ids[0]' = inspection_fields$id
)
res <-
httr::VERB(
"POST",
url = "https://api.iasset.nl/getCustomInspections",
body = body,
encode = 'multipart'
)
# cat(content(res, 'text'))
res %>% httr::content()
get_records()
takes a while to run, this is because it currently does two requests including a big one to actually fetch all the records in one go. With filtering #16 we'll need another request.
verbose = TRUE
argument to keep the user up to date on what the function is doingIdeally we'd also add a vignette on how to use the package, but this is something for beyond v0.1.
Things to mention in the readme:
get_access_token()
for a token, get_records()
to get records and get_fields()
to get information about the fieldshis orcid: https://orcid.org/0009-0003-2454-8521
The failure state when you enter the wrong username isn't very elegant:
get_records(access_token = get_access_token(username = "[email protected]"))
#>Error: access_token is not a string (a length one character vector).
I'am experiencing issues with getting get_records()
to run with github actions. I get the following error:
Error in b__file_keyring_autocreate(self, private, keyring) :
The 'system' keyring does not exists, create it first!
Calls: source ... b_file_list -> <Anonymous> -> b__file_keyring_autocreate
Execution halted
in https://github.com/inbo/aspbo/actions/workflows/get_vespa_velutina_management.yaml. My latest run: https://github.com/inbo/aspbo/actions/runs/8615691797/job/23613099659
its seems the error occurs when the workflow runs the following code:
https://github.com/inbo/aspbo/blob/afee4e3527ef617f9a1c6074242b838989d9ed71/src/Vespa%20velutina%20management/get_data_from_iAsset.Rmd#L33-L39
When I look at the keyring readme I see the following remark:
The envvar backend doesn’t support custom keyrings, so if you’re using one locally you’ll need to use the default keyring on GitHub.
This makes me suspect adding service = "iasset_password"
is the cause of this issue.
PS I added "iasset_username" & "iasset_password" as github Actions secrets on the repository level.
Not very difficult with cffr::cff_write()
but I'd prefer a solution that will automatically update the citation.cff with every release
Currently we have a few multivalue in the API output, we still need to handle these more elegantly before we can output to a tibble.
Currently we have explored tibble::enframe()
, and tidyr::pivot_wider()
res %>% httr::content() %>%
purrr::pluck("returndata") %>%
head(5) %>%
purrr::map(~purrr::pluck(.x, "data")) %>%
# purrr::map(~purrr::map_dfc(.x, function(element){element}))
purrr::map(~tibble::enframe(.x)) %>%
purrr::map_dfr(~ tidyr::pivot_wider(.x, names_from = name))
But I personally think it should probably be handled with a combination of purrr::map_dfc and purrr::map_dfr
Currently we return this as an integer, encoded as a character. There are only a few values in use, what do they mean?
tijd_ter_plaatse | n |
---|---|
10 | 4 |
120 | 1 |
1215 | 1 |
15 | 8 |
20 | 5 |
30 | 3 |
40 | 1 |
45 | 1 |
5 | 3 |
60 | 3 |
90 | 1 |
NA | 9489 |
It would probably be more user friendly to return these values either with a unit, or as a lubridate::duration()
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.