Add occurrence count to metrics about gbif-dataset-metrics

[bartaelterman]

@niconoe can you add a NUMBER_OF_RECORDS to your output?

I'll aggregate everything from one dataset together and compare the total sum with the API.

@peterdesmet how important is it that these match? i.e. what should happen if these numbers are not equal?

[niconoe]

Yep, will include NUMBER_OF_RECORDS in first iteration of the data_extraction_module.

[peterdesmet]

@bartaelterman, then the percentages would be too low.

[bartaelterman]

The API might not be a good idea after all. Since the plan is to do this in the backend scripts, that means it will be done for every dataset we index. The goal is: all of them. That's about 14.000 datasets and thus 14.000 API calls we'll be sending to GBIF. We can do stuff in parallel with requests asynchronous support but we'll have to set a limit to the number of calls to prevent being blocked.
Personally, I would leave API calls for the front end only, because then you only do calls for the dataset you're viewing.

The total number of records is still valuable, but we'll just get it from the metrics, and don't perform a check. (unless you really insist, but then I would do that in the front end).

[bartaelterman]

This does bring me to the following issue:

Since we're using chunks of the download, and I don't know how these were cut, it's possible that the data for one dataset is spread over several chunks. If not, then my point will not be valid, but then there would also be no use for aggregating data coming from different chunks. So I assume it does happen.

The problem is, we never know when we have all the data for one dataset except when we have processed all chunks. The only other option is what @peterdesmet suggests, but that is challenging because of the sheer number of API calls. The other option would be instead of working with one big download and cutting it ourselves, working with smaller downloads because then we know that the datasets in one download are complete.

[niconoe]

My plan to circumvent this issue (ensure we have fully covered a given dataset) was:

• First step: working with only a few selected datasets, 1 archive per dataset.
• Later, split the whole archive and process ALL chunks.

[bartaelterman]

Ok @niconoe . Then I would leave the check with the API call.

[peterdesmet]

OK.