jncc / datahub Goto Github PK

Upgrade the PoC Search page to use the live ES index. Will have to solve authentication between the Beanstalk app and ES.

What database shall we use?

There are several different types of resources or datasets:

• Open Data datasets (with or without a DOI)
• Publications
• Scientific documents
• ... what else?

The search results need to paginate.

test other document types to check performance

Proof of concept Search / Search Results with basic demo keyword filter by querystring.

static entries with links to individual resource pages

For some reason the NamedProfileCredentialProvider in ElasticsearchService doesn't seem to work.

• Is a new "website" Topcat category feasible?

Some abstracts are written in Markdown. We could support rendering the markdown as HTML.

Test and Live servers.

Update the lambda function to upsert the record into an elastic search index.
Includes create elastic search instance.
Check can do basic full text search.

Meeting for Friday morning - summary of options.

Matt has a new deployment mechanism to replace Felix's - but it needs to be automated.

The full steps to manually deploy are now in the readme.md but this is still a manual process.

Populate with dev data

Questions:

• Can we migrate from data.jncc.gov.uk to S3 later without changing URL?

These questions are now less relevant:

• Can we use Defra ongoing?
• Is there an easier way to extract things from glacier?

The home page is a search page.

• Search box
• Count of datasets (and other statistics?)
• Display latest datasets

...in format from the catalog/inventory and filter (note tweaking needed for multiple resources not viewable in UI, include DOI record- type doi into search to bring it up)

Design notes / discussion for the main Datahub Asset class.

Common Standards Monitoring

• What are CSM collections for?
• Are CMS data assets reports? Currently they're 'series' - how are genuine series going to be handled by the data structure?

General design

• Data Format is currently missing

We just need single keyword search by URL and from the asset page keyword, for starters.
Adjust search page to show that it's a keyword search.

It would be nice to have the link to data.gov.uk page if it exists on our resource / asset landing page.

This could be tricky.... The current plan is to (logically at least) publish to the Datahub first, then DGU.

• Is the data.gov.uk URL conventional or calculable?

Needs investigation. Could fundamentally affect the publishing mechanism design.

The 'Show more' / 'Show less' implementation was a fairly quick CSS-only solution for the proof-of-concept to long abstracts, and doesn't work very well if the abstract is shorter than 10 lines - the 'Show more' button always shows, and there's excessive whitespace.

This is a definite problem for some records with shorter abstracts.

Plan is to use the main website page. Link to from the footer.

We don't think there' anything else to do really.

The colour hashing code from Topcat needs translating from Coffeescript into C#. There's some subtleties around 'vocabless' keywords too.

creates template and prints out page based on page Topcat ID

Use ideally Shippable to push a build on commit.

Tetrienne and MNR staff are sending round to existing users a copy of their privacy policy as part of GDPR readiness.

They need a list of emails to send the message to - individuals and organisaitonal emails

Please extract from DB as CSV / Excel and sent to Tet.

The website needs revamping which involves reducing the number of webpages. The data, such as publications and scientific documents, from these pages need to be stored, viewable, and searchable by the public from one central place.

https://jncc-my.sharepoint.com/:w:/r/personal/lynn_heeley_jncc_gov_uk/_layouts/15/Doc.aspx?sourcedoc=%7B8D2CC1E3-7239-4A5B-9B1F-E94A06F4D9F1%7D&file=Data%20Hub%20Requirements%20v2.docx&action=default&mobileredirect=true

How (and should) we do this in the Search page URL for linking to from e.g. website pages?

Need to understand more about how Elastic Search works, whether this is possible, and then whether this is desirable.

The alternative is to have our own search parameters much like Topcat.

We don't want to appear to accidentally claim ownership of something which isn't ours. The Topcat label of Responsible Organisation / Author is not appropriate. Something like:

Organisation
JNCC Distributor
[email protected]

We need a queue to handle the website republish.

Make an HTTP API which accepts a JSON record and upserts it into a Dynamo table.

Implement the Search page.

Complete the implementation of a command line tool for Creating, Populating and Deleting index entries.

• Command line app
• Ingest PDF examples
• Working both locally and with signed requests on AWS

PDF search needs more 'proving' asap.