nceas / arcticreport Goto Github PK

should look like this, need it quarterly

The way we count creators right now is not ideal - just looking at names. Are there better ways to do this? ORCIDs are an obvious choice, but much of the legacy data doesn't have ORCIDs.

The two primary query functions in the package: query_objects and query_version_chains take 20 minutes and 100 minutes to run, respectively. query_objects returns a data.frame with a row for every object in the ADC. query_version_chains takes the result of query_objects and assigns an arbitrary series identifier to each version chain. The rest of the functionality in the package is slicing, dicing, summarizing, and plotting metrics based on those two tables.

Since the functions take so long to run, it is definitely not practical to run these two functions often. For CI, we could build a status page that runs everything once a day or so, and fills in tables for the quarterly metrics when those milestones show up. For local testing, or creating one-off kind of plots, it would be beneficial to set up a standard way of caching those query results for ease of use.

Open to any suggestions. The bigger of the two tables is about 100MB when saved to disk.

It works in certain cases, and you can plot cumulative count or size of either data files or metadata files, but it could be smarter at:

• units (for the size metric)
• axis labels
• where in time the plot starts
• placement/size of the ADC start line

The unnesting step in this function (tidyr::unnest_longer) is fairly slow, performance I think could be improved by using data.table instead, as shown here. Only downside is that it adds a dependency

I had the idea on a call that there is a way we can make query_objects much faster by keeping parts of the cache that are still relevant. Below are some changes that would need to be made

• add dateModified to the fields returned by the query function
• if a cache is found, filter to keep all objects with a dateModified older than the datetime at runtime
• query for objects only with a dateModified more recent than the datetime at runtime
• save the cache
• if no cache is found, then make sure the datetime you are querying against is very far in the past so that you get all of the objects
• remove the cache_tolerance parameter

the count_creators function has a list of creators that are removed. The comment in the code says:

# Grep-based filters
# Bryce created these (and we can expand these) based upon what I saw in the results
# that looked like organizations or non-persons of some sort or another

We should review this list against the list of unique creators and decide if we want to expand, revise, or altogether remove this list.