sanger / biosero-sql Goto Github PK

ToDo's

BioSero Technical Meeting 21st May

Harry redo schema diagram

• DBA meeting on Monday

1. Look over, take through process/ talk through what schema tables represent in "real world"
2. What permissions are required for BioSero to access stored procedures? Read-only to config table? And how to create a stored procedure
3. Provide estimations of size for scalability (30,000 samples a week - source/dest, 1/100 ~300 runs a week, ~600 control-plate-well a week). Automated data creation for testing 6 month/ 12 month?
4. Transactions, level of visibility during transaction process, recommended "isolation level" in database configuration.

• Events table in central database
  Check in Friday technical meeting with R&D what is to be written in this table
  Decide where errors should be added to events from process
  Rename table to not clash with events?
  Discussed:
  Table name could be 'automation_system_events', or 'run_events'.

• Action: Andrew to change table name to run_events
  R&D to consider usefulness of events other than errors being recorded.

• Discuss Systems state end point - what exactly to write in URS? Will Sanger add an API endpoint to micro webservice on central database to fetch states from both systems and write to table for Tableau to use (heartbeat)?

Discussed:
A 'nice to have' for R&D. Scott could derive it from the run states.
Are run state and system state equivalent? Nearly. Rob offers to update state in the run table whenever it changes, but it would be within the context of a run.
URS should retain requirement for Biosero to provide system state RESTful API endpoint to fetch state.

• Discuss how we're going to handle BioSero interaction with the central database
  Access: via API on web service, stored proc, directly?
  Permissions required: RO/ RW etc for each table. e.g. configurations permissions RO?
  Ownership?

Discussed:
Rob expresses a strong preference for creating stored procedures to access tables. Happy to collaborate on the development of those stored procedures.

• Action: all, create stored procedures

• Action: @pjvv to add Rob as a collaborator to the MySQL github repo
  Biosero seem happy to own the data and not the schema / hosting of the database.

• Action: We can remove table drop/create from Biosero accounts. They will need to be able to access stored procedures.

• Action: Discuss permissions on accounts with DBAs Monday

• Discuss how to handle failure to connect to central database. What if picked some wells and cannot record them? Retry? If cannot reconnect then what? Abort and log locally? How to fix data inconsistancy?

Discussed:
Various error scenarios are possible. Rob and John to go through Hamilton scenarios.
Hamilton does update various files as it goes so state of plates should be possible to determine.
Likely a need for a SAT test to check error recovery process.

• Action: Andrew to add placeholders for extra use case in URS doc and test in SAT doc for Recovery process

• Follow up on BioSero API document

Discussed:
Biosero API docs not ready yet.

Other

• Add new story to back log for Microservice
  Design and plan service to sit in front on central database
  Along with views/ stored procedures/ query to access current state of play

• Review back log stories

• Remove all table references from URS

• Tidy up biosero-sql repo
  README etc
  Intuitive folder sturcture

• Perform optimisation tests on DB
  Check we have correct indexes
  Check tables are performant with lots of data

• API design
  Simplified - send minimal data, use central database as source of truth

• Add fields to config for bed verification
  barcode is physical stuck on plate position on the deck
  the users scan deck location and the plate location to verify type of plate is in the correct place
  will need to know expected deck location barcode
  e.g bv_deck_barcode_control
 bv_deck_barcode_destination

• Standardize all names in config table
  for ease of searching e.g 

select * from 
where config_key is plate_barcode_prefix_%
  e.g plate_barcode_prefixes_<plate_type>

• Pull docs out to repo
  Move SQL queries out from URS

• Create Schema diagram @harrietc52
  Go into reverse engineer/ database/ biosero

• Add enums to ERD and SQL
  mysql8

• LIMS api schema doc
  Add to URS
  Phil and Eduardo updating

• Constraint on source/ control from destination wells table @harrietc52
  use check
  add constraint that both source_plate_well_id and control_plate_well_id can be null, but only one can be populated

CONSTRAINT `source_plate_well_id_or_control_plate_well_id` CHECK ( `control_plate_well_id` = 1 XOR `source_plate_well_id` = 1 )

• Check control_plate_wells is referenced correctly in URS

Parked

• Adding config to support excluding all from destination coordinates
• Process diagram for Lighthouse UI