sanger / cherrytrack Goto Github PK

User story
As Biosero we would like a stored procedure we can call that returns the summary information for a run. For use in creating the summary display panel for the users at the end of a run.

Who are the primary contacts for this story
Rob H
Andrew
Eduardo

Acceptance criteria
To be considered successful the solution must allow:

• stored procedure is created getRunSummaryInformation
• returns list of controls and source plates used in the run with counts of wells picked (in this run)
• for each source plate the count of empty wells remaining (zero for fully picked, > 0 for partially picked)
• for each source plate whether it was partially picked in any previous run(s) (a boolean) i.e. samples picked into other destination plate(s)

Dependencies
This story is blocked by the following dependencies:
None

References
This story has a non-blocking relationship with:
NA

Additional context
Difficult to achieve this in one query for a view.

User story
As R&D we need a way for lab users to view the current state of destination and source plates in the Cherrytrack database for use after errors occur.

Who are the primary contacts for this story
Scott T
Lesley S
Andrew
Eduardo

Acceptance criteria
To be considered successful the solution must allow:

• the user to scan or enter the source or destination barcode into a lighthouse ui screen
• display of a summary screen for the information for the scanned plate including numbers of filled and empty wells and information for each well and it's contents

Dependencies
This story is blocked by the following dependencies:
This story will need stored procedures to fetch the information for a barcode from cherrytrack, a screen in lighthouse-ui to enter barcodes and display summary information, and an interface in lighthouse for the ui to call to fetch the information from cherrytrack. Possibly endpoints in cherrytrack to encapsulate the information retrieval.

References
This story has a non-blocking relationship with:
NA

Additional context
This story was identified during SAT testing. For Beckman this exists in the form of the Beckman DART web interface. The lab users use it to diagnose what has happened to plates after an error, e.g. whether a source plate still has samples to pick, or whether a destination plate has empty wells remaining, or which samples were picked into what wells.
Need to discuss with Scott the format of the summarised information. Fastest would be to just list by well as text similar to Sequencescape plate summaries. A fancier solution would have a view of the plate (like in Limber) with colours indicating filled and empty wells.

Describe the bug
As R&D we would like to move a plate created in UAT into Production. The first Biosero run intended to be in production was actually done in UAT by mistake (the configuration change didn't save). The run completed cleanly.

RT Ticket Number
728773

To Reproduce
Steps to reproduce the behaviour:

1. GBG exe config on the Biosero PC has 2 sections for UAT and Prod connection details

Expected behaviour
Permissions issue with file had prevented the file saving.

Screenshots
NA

Desktop (please complete the following information):
Biosero Windows PC

Additional context
UAT run 240 for destination plate HT-117470 and source plate cpRNA104065 needs to be transferred into production.
See google doc https://docs.google.com/document/d/1O6lVajlwOlcXacTjMXPpSMW4no62NFhW2PsFp8SBMBM/edit?usp=sharing

Description
There is currently a problem where lab staff will occasionally attempt to pick the remainder of a partially picked source plate on a different system type. e.g. pick partially on Beckman, try to complete on Biosero, or vice versa.

This limitation is clearly understood by R&D and Operations, and is strictly forbidden in SOPs and training, but is still happening.

Because the two picking systems use different databases (Biosero uses CherryTrack, Beckman uses DART), there is nothing to prevent picking attempts for the same wells on both systems. This generates an error only at the end of the run when attempting to create the destination plate in Sequencescape, due to sample uuids already existing from the first pick. There is nothing the lab staff can do to fix this error.

This results in support tickets for PSD to manipulate the data. It also risks barrel contamination on the robot systems when they attempt to aspirate previously picked wells and instead create aerosols.
Example RT: https://rt.sanger.ac.uk/Ticket/Display.html?id=735700

The intention was to eventually combine cherrypicking into one database, so there was one source of truth. This would be a large task however.
This research story is to see if there is a quick way to prevent the above issue, by either filtering the pickable wells list or throwing an error earlier in the run.

Who the primary contacts are for this work
Scott Thurston
Sara Stott

Knowledge or Stake holders
Andrew
Eduardo
Harriet

Additional context or information
Options:
Biosero source plate lookup API endpoint in lighthouse could check DART and filter the returned list to remove already picked wells.
On picking a source on Biosero we could trigger an update of well parameters in DART so no longer set to 'pickable' for Beckman.
Use events to trigger updates in the opposite system?

To consider:
Affect on picking reports.
Partial destinations, when do the source picks get recorded on Beckman.
Could a different lab process solve this issue? e.g. isolate sources earmarked for Beckmans so can never be used on Biosero and vice versa. (Scott)

Describe the Housekeeping
Security - BioSero -> Cherrytrack DB (check biosero DB permissions on stored proc or specific tables in DB - psd_admin has GRANT)

Blocking issues
Describe any other issues or tickets that may be blocking this change.

Additional context
Add any other context about the problem here.

Describe the bug
R&D have seen a number of 'lost connection to MySQL' errors during the testing of the Biosero systems in UAT. The DBA team (David H) have told us there is a known issue with v18.04 of Ubuntu whereby it drops a connection during a refresh of the DHCP lease. Eduardo attempted applying a fix to v18.04, but we have seen a further instance of the error (31st Aug 11.34am). The solution according to the DBAs is to update the Ubuntu version from v18.04 to v20.nn
A retry on the step in the GBG software allows the user to recover and continue.

RT Ticket Number
730139

To Reproduce
Not easy to reproduce, seems to occur very rarely. DHCP lease is only updated once in 3 days according to DBAs. And has to coincide with MySQL query in progress.

Expected behaviour
DHCP lease renewal should not affect queries.

Screenshots

Additional context
Risk when upgrading OS that applications will not come back up. Have plan to revert back to v18.04 and test on UAT and Training before Prod.

User story
This is a research story to identify the tasks required to upgrade Ubuntu to 20.nn. The End of Life for v18.04 is 30th April 2023

The primary requirement is captured as part of the #36 ticket which can be converted and split into smaller user-stories.

Who are the primary contacts for this story
e.g. John S (don't include surnames in public repos)

Who is the nominated tester for UAT
e.g. John S (don't include surnames in public repos)

Acceptance criteria
To be considered successful the solution must allow:

• Attached a list of servers that needs an Ubuntu upgrade.
• Trial upgrade of any server to look for issues and downtime(Gatekeeper training upgrade testing done).
• Created list of servers to be upgraded.

Dependencies
This story is blocked by the following dependencies:

• #<issue_no.>
• sanger/#<issue_no.>

References
This story has a non-blocking relationship with:

• #36
• sanger/General-Backlog-Items#284
• sanger/General-Backlog-Items#285

Additional context
Add any other context or screenshots about the feature request here.

User story
As Biosero we would like to be able to warn the user at the start of a run when a destination plate has already been created in the LIMS (e.g. partial created as result of error recovery or created via lighthouse-ui).

Who are the primary contacts for this story
Rob H
Andrew
Eduardo
Harry

Acceptance criteria
To be considered successful the solution must allow:

• add a new table 'destination_plates_in_lims' with a barcode (unique) column
• add a new endpoint for lighthouse to call with a barcode
• endpoint adds the barcode to the table
• add a new stored procedure 'isDestinationPlateInLIMS' that takes a barcode as input, checks the new table, and returns a boolean (see existing stored procedure 'doesSourcePlateExist' for example

Dependencies
This story is blocked by the following dependencies:
None

References
This story has a non-blocking relationship with:
Lighthouse story sanger/lighthouse#397

Additional context
Discussed error recovery at technical mtg 16th July. Outcome of that was that this would be good to have in place.

User story
As a dev, would would like to create a microservice called <....> to sit infront of the Central DB, providing certain endpoints to aid Lighthouse to get the data required for different events

Who are the primary contacts for this story
Andrew S
Phil JV
Harriet C
Eduardo

Acceptance criteria
To be considered successful the solution must allow:

• Automation system run endpoint @harrietc52
• Source plate wells endpoint @harrietc52
• Destination plate wells endpoint @harrietc52

Additional context
Biosero Micro Service

Describe the Housekeeping
As a developer (ay6) I would like cherry track dependency updates enabled as GitHub workflows do not work upon merging a PR to develop branch.

Acceptance Criteria

• GitHub workflows authorisation issue is resolved
• Dependency updates are applied (dependabot PRs)

Blocking issues
No blocking issues.

Additional context
I think the CI fails because of configuration issue in authorisation of dependabot or the user triggering the workflows.

As R&D we would like to change the location of Heron cherrypicking controls on the Biosero system to enable the transition to strip tubes and automated control preparation.

From Biosero Technical meeting on 30th July - Update Cherrytrack config once SATs have finished. We will need to coordinate the change in config with Biosero (Rob/ Scott?)

Acceptance Criteria:

• Cherrytrack ‘configurations’ table should increment to a new version and implement the following changes from v1.0
  * Control_coordinate_positive = D1
  * Control_coordinate_negative = A1
  * Control_excluded_destination_wells = A3, B3, D3, H3, H10, E11

Coupled with sanger/sequencescape#3272

Contacts
Scott T

User story
As developers we would like to have an intergration E2E testing supporting the Biosero process, simulating SAT 9

Involving:
Lighthouse UI > Lighthouse > Baracoda > Crawler > CherrytrackDB > Mongo > MLWH > Limber > SS

Who are the primary contacts for this story
Harriet
Andrew
Stuart
Eduardo

Acceptance criteria
To be considered successful the solution must allow:

• Create new test suite in Integration suite project (dont edit existing Heron LTHR)
• Creates the testing run data (via LighthouseUI UAT Actions)
• Replicate the basic robot run
  check correct events are fired
• To create correct data in Cherrytrack DB
  check run exist
• To create correct data in Lighthouse MongoDB
• To create correct data in MLWH
• to be able to create a workflow in Limber or SS

Additional context
Biosero architecture

Existing python script to simulate a Biosero run

Describe the Housekeeping

Test existing endpoints

• /beckman/failure-types
• /beckman/robots
• /cherrypicked-plates/create
• /cherrypicked-plates/fail
• /plate-events/create
• /plates/new

Required for SAT

• Rename EVENT_DESTINATION_CREATED -> completed
• Remove all things EVENT_SOURCE_ALL_NEGATIVES
• update generation of testing data script to pull picked samples from mongo (@andrewsparkes )
• Update PostMan (Lighthouse endpoints)
• Add index samples collection in mongo for lh_sample_uuid
• check events message include user/robot? NOT NEEDED, REMOVED
• Remove robot and user id from event parameters
• Merge cherrytrack changes into develop and deploy to UAT
• Deploy lighthouse dev code branch into UAT
• Drop and recreate UAT database from cherrytrack
• Integration test run to check all working

Post SAT todo's/ Required for LIVE:

• Add lh_sample_uuid Index to PROD mongo samples_collection (add to Crawler) ready to deploy @andrewsparkes
• Make event names consistent - then email Rob (and requires URS update) - decided against this on 15/6
• Add robots as users to labwhere in PROD
• Add protection to reset script - DANGER to prod (PR here)
• Add the call to Labwhere to set source plates sent to output stacks for specific event types = source_completed and no_pickable_samples

Nice to Have:

• Security - BioSero -> Cherrytrack DB (check biosero DB permissions on stored proc or specific tables in DB - psd_admin has GRANT)
• Integration test - E2E via Docker (post SAT)

DONE

• Backups with DBA (Done: email David Harper re: backups / frequency and replication. Todo: Rich L to confirm about Tier 1) @andrewsparkes
• Security - Lighthouse -> Cherrytrack (GET only - for first release just use HTTPS, but no API key) @pjvv
• Security - BioSero -> Lighthouse (POST events HTTPS, including API key per env) add api key to lighthouse (flask default behavior)
• Deployment DPL-023-1 @pjvv
• Update PostMan (Cherrytrack endpoints)
• Self contained deliverables (docker-compose.yml in Cherrytrack) @pjvv
• Write access tests from BioSero to the DB

User story
As a PSD developer I want to add a unique constraint for barcode and location in destination_wells so we don't allow duplicated wells in the same position across different run_ids

Who are the primary contacts for this story
Eduardo Martin
Harriet Craven
Andrew Sparkes

Acceptance criteria
To be considered successful the solution must allow:

• Block any attempts to create a duplicate well in the same position for a barcode

Dependencies
This story is blocked by the following dependencies:

• #<issue_no.>
• sanger/#<issue_no.>

References
This story has a non-blocking relationship with:

• #<issue_no.>
• sanger/#<issue_no.>

Additional context
Add any other context or screenshots about the feature request here.

User story
As PSD and R&D We need a system to regularly poll the API endpoint for each system and write the status in the Cherrytrack database so that we can remotely look at the status of the Biosero systems.

Who are the primary contacts for this story
Scott T
Andrew
Eduardo

Acceptance criteria
To be considered successful the solution must allow:

• scheduled API call to fetch status for each system (every 1 minute?)
• new table 'automation_system_status' in cherrytrack database to store system status (no history?) (automation_system_id, status, updated_at)
• table accessible to Tableau

Dependencies
This story is blocked by the following dependencies:

• Biosero implementation of GBG API, currently WIP by Rob H on 10th Aug

References
This story has a non-blocking relationship with:
NA

Additional context
This story does not describe or include any dashboard or use case for displaying the statuses. Needs discussion with Scott.