Dataset Submission: HakaiQuadraBoLResearch about hakai-datasets

[JessyBarrette]

CKAN record available here on pac-dev1
https://pac-dev1.cioos.org/ckan/dataset/ca-coos_4d1080c0-c9e5-4551-a463-378040ccfe20

[JessyBarrette]

The development dataset is now available on the hakai development erddap server:
https://goose.hakai.org/erddap/tabledap/HakaiQuadraBoLResearch.html

[raytula]

Hi @JessyBarrette I'd like to take a crack at pushing this into to the sensor network database. Please point me to the best place to find a version of the data that may be relatively simple to parse and push into the SN database, similar to what I'm doing with our other standalone data. Based on looking at the data via ERDDAP, it may be pretty simple to do. Thanks

[raytula]

CKAN record available here on pac-dev1
https://pac-dev1.cioos.org/ckan/dataset/ca-coos_4d1080c0-c9e5-4551-a463-378040ccfe20

I don't see Wiley referenced anywhere there. Should/could he be?

[JessyBarrette]

I don't see Wiley referenced anywhere there. Should/could he be?

Hum that's good question regarding Wiley, I'll look back at the form see if it's name is there. I believe it is.

Please point me to the best place to find a version of the data that may be relatively simple to parse

Maybe the best is to use the erddap dataset directly? Otherwise, I can either give access to the raw data through ERDDAP.

[JessyBarrette]

That's interesting Ray regarding Wiley's contacts are missing. Looks like there's a bug in the transition from the form to CKAN. Here's the related form: https://cioos-siooc.github.io/metadata-entry-form/#/en/pacific/tV5qE0aUgaOjSVmgPgiZ6MyHuSy1/-MVOSTlh_rgekISRw3Q8

Wiley was the only one that had the organization fields filled and for some reason, we would only see on CKAN the organization and not the author. While the other had just their name. I made some changes, but I think @n-a-t-e should have a look

[n-a-t-e]

I will pass this one to @fostermh , Matt- Wiley's name is getting to /mdb:MD_Metadata/mdb:contact/cit:CI_Responsibility/cit:party/cit:CI_Organisation/cit:individual/cit:CI_Individual, Maybe this location isnt getting displayed in CKAN?

[fostermh]

hmm yep, it was broken. nice to see the individual nested under the organization. Apparently, I had not updated the xpath's to handle that. I will push a fix and update dev tomorrow.

[fostermh]

sorted on dev.
https://pac-dev1.cioos.org/ckan/dataset/ca-coos_4d1080c0-c9e5-4551-a463-378040ccfe20
and
https://pac-dev1.cioos.org/ckan/dataset/ca-coos_a6f6cd81-f387-4915-a690-8ac8d21a1419

[raytula]

Thanks @fostermh I'm not sure if this is a you thing, an @JessyBarrette thing and/or an @n-a-t-e thing, but I don't find the links to the SOCAT metadata files or data submissions particularly user friendly. I was not able to actually view any of them in practice. I'm not sure if these are intended to be links to the actual source (ie. SOCAT), stored locally or ...

[JessyBarrette]

Nice catch @raytula, that's a me thing I need to update the link to those files hosted on the github repo!

[raytula]

Nice catch @raytula, that's a me thing I need to update the link to those files hosted on the github repo!

Could/should we link to the same or similar content hosted at SOCAT?

[JessyBarrette]

Yeah, we could totally also add a link to the SOCAT. I would need to dig a bit to find them again.

The documents were uploaded to the main branch of this repo:
https://github.com/HakaiInstitute/hakai-datasets/tree/master/datasets_documents

And the links updated on the metadata form:
https://cioos-siooc.github.io/metadata-entry-form/#/en/pacific/tV5qE0aUgaOjSVmgPgiZ6MyHuSy1/-MVOSTlh_rgekISRw3Q8

Not sure how long it takes to get reflected on the pac-dev1 CKAN

[fostermh]

datasets updated. link's still seem to be broken. get a 404 when clicking on the link.

you probable want links that look like https://github.com/HakaiInstitute/hakai-datasets/blob/master/datasets_documents/HakaiQuadraBoLResearch/Hakai_OAP_metadata_QuadraBoL_Apr2016_Dec2017.docx not the /blob/ followed by the branch name, in this case master

[JessyBarrette]

Hum it would work yesterday, ok I'll change those. Thanks @fostermh !

[JessyBarrette]

Hi @JessyBarrette I'd like to take a crack at pushing this into to the sensor network database. Please point me to the best place to find a version of the data that may be relatively simple to parse and push into the SN database, similar to what I'm doing with our other standalone data. Based on looking at the data via ERDDAP, it may be pretty simple to do. Thanks

@raytula We've added an access to the source files of each OA ERDDAP datasets:
https://goose.hakai.org/erddap/files/HakaiQuadraBoLResearch/

You could potentially use those but they may change over time, I feel like using the ERDDAP dataset itself would be better since it is less likely to change over time. The original files may change based on what the OA group will provide us in the future. (ideally it would the same but we can't really be sure.)

[JessyBarrette]

@fostermh I replaced the section "blob/master" by "raw/master" in the link to the documents on Github so that by clicking you get directly access to the file itself and not he Github page where it's hosted. Let me know if that works for you. The link aren't updated yet on CKAN, @n-a-t-e was gonna have a look.

[JessyBarrette]

@raytula documentation was replaced by a link to the NCEI dataset. I also saw that newer data was available on NCEI and updated the ERDDAP record with the newest data.

Update to the form should be available on CKAN soon @n-a-t-e or @fostermh will be able to tell.

Here's a link to the updated metadata form https://cioos-siooc.github.io/metadata-entry-form/#/en/pacific/tV5qE0aUgaOjSVmgPgiZ6MyHuSy1/-MVOSTlh_rgekISRw3Q8

[raytula]

Thanks @JessyBarrette the updated link to NCEI is good.

Good to see and know that the NCEI dataset seems to stay the same as new data is added.

A few things I noticed, some of which may still be already known:

• The CKAN dataset title still shows 2019-12-31 as the dataset end date. This should be changed or, possibly preferred, remove the end data from the dataset title
• The dataset abstract includes an encoding issue or something like that (Hakai Instituteâ��s)
• The referenced ERDDAP resource does not exist
• The list of responsible organizations is empty (see below)

[JessyBarrette]

Yes I agree that having no end dates within the title and summary would simplify the setup.
Todo

• Remove dates from title/abstract:
  • metadata form, There's a bug with the metadata form and we can't yet update the dataset title
  • ERDDAP dataset.xml
• HakaiQuadraBoLResearch available on Goose ERDDAP
• Responsible organization
• Remove encoding issue within the summary

[fostermh]

move

Name: Wiley Evans
Affiliation: Hakai Institute
Contact Info Email: wiley.evans(at)hakai.org
Role: Custodian

from point of contact to responsible parties will fix the resp. org. issue and also correctly represent what is going on in the metadata.

point of contact is for 'point of contact' only. If you really want to have a 'custodian' role in there you will also need to add it to the responsible parties list.

[JessyBarrette]

Got it, thanks Matt!

…

On Wed, Mar 10, 2021 at 10:23 AM Matthew Foster ***@***.***> wrote: move 'Name: Wiley Evans Affiliation: Hakai Institute Contact Info Email: wiley.evans(at)hakai.org Role: Custodian' from point of contact to responsible parties will fix the resp. org. issue and also correctly represent what is going on in the metadata. point of contact is for 'point of contact' only. If you really want to have a 'custodian' role in there you will also need to add it to the responsible parties list. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHICYONY2MN45KTFVSHCWFLTC6TKZANCNFSM4XDHOVUA> .

-- *Jessy Barrette M.Sc.* *Marine Instrumentation Specialist* Hakai Institute <https://www.hakai.org/> | [email protected] | (C) [phone]

[fostermh]

getting closer.
https://pac-dev1.cioos.org/ckan/dataset/ca-coos_fe76ef4c-254a-44fe-87bc-052cd3aa9663

Note the new 'citation' box

Are the dates correct? what I mean is are all these dates related to the metadata as the date field indicates or are some 'data' related dates? I think I had a pr to change the way ckan handles this but not sure if it's merged yet. also not sure if the metadata form has been updated to put dates in two spots yet, one for metadata, one for the data. @n-a-t-e?

[raytula]

Note the new 'citation' box
Looks great!

Jessy and I will be going over everything with Wiley next Thursday morning, so hoping to have everything in the final/production-ready state by then, and hopefully push into production soon afterwards.

[JessyBarrette]

@raytula I think we're getting pretty close to a presentation level with this dataset. I think the only issue left is to change the Metadata Record title to avoid having any dates in it. There are some issues with that right now hopefully this will get sorted out at some point soon, @n-a-t-e is on it. But I'm wondering if we should also leave the NCEI Accession number within the title?

If yes it would be better to do the same with the two other OA datasets.

Those changes may not happened before the title change issue is resolved on the metadata form.

[raytula]

I recommend you remove the NECI Accession number from the title

[JessyBarrette]

@n-a-t-e said that the title issue is over, I changed the titles to the finalized version. Should be updated on pac-dev CKAN shortly!
https://cioos-siooc.github.io/metadata-entry-form/#/en/pacific/hM2aN3v9MzZ6J4KGMOEW2s9gidj2/-MVOSTlh_rgekISRw3Q8

[JessyBarrette]

Finalized datasets for presentation to the data administrator:

• dev ERDDAP
• dev CKAN
• CIOOS Metadata Form

[raytula]

Thanks @JessyBarrette . The CIOOS metadata link is linking to 'Not found'

[JessyBarrette]

@raytula Thanks just updated it. @n-a-t-e made some changes to the cioos metadata form website some old links may not work anymore. He said he corrected most of the links shown on Github but there's maybe a few left. I'll have a look at the others.

I think I will also move those links to the first issue comment

[JessyBarrette]

This dataset is now available online. We can close that issue!