sanger / tol-lab-share Goto Github PK

Description
Complete a schema definition (which defines which data will be sent), approved by ToL and SciOps by Oct

Who the primary contacts are for this work
Eduardo (PSD)
Steve (PSD)

Knowledge or Stake holders
TOL, SciOps

Additional context or information
Agreed standard for sharing sample data between faculty and operations

Description
As PSD developer I want to agree with TOL the next stage of the development to provide a consumer to ingest samples from TOL into Traction because I want to coordinate any required changes in both sides to perform the coding of the solution.

Who the primary contacts are for this work
Eduardo Martin

Knowledge or Stake holders
Sujit Dey
TOL

Additional context or information
Any other useful context or information that may be useful.

Description
Establish a shared understanding of the process of sample ingestion between TOL - SciOps - process maps agreed with ToL and SciOps by Oct.

Discoverable process which can be viewed by all stakeholders as an agreed standard of how to request work

Who the primary contacts are for this work
Eduardo
Steve

Knowledge or Stake holders
TOL, SciOps

Additional context or information

Enabling Platforms & Core labs (Progs: ToL), SciOps - Long read, Sample Management, SSRs

User story
As PSD we need to modify the ToL Traction Consumer so that it can navigate the response to the correct source. (Consumer currently supports messages only from TOL, but it will also need to support messages from Limber/SS).

Who are the primary contacts for this story
PSD
Eduardo MJ

Who is the nominated tester for UAT
PSD

Acceptance criteria
To be considered successful the solution must allow:

• Import of SS/Limber tubes into Traction via the consumer
• Response (feedback) to be routed to the correct queue for SS/Limber consumption

Dependencies
This story is blocked by the following dependencies:

• None

References
This story has a non-blocking relationship with:

• x

Additional context

User story
As a TOL user I want to add genome size and accession number to the message so it can be sent to be used by Traction downstream process

Who are the primary contacts for this story
Stephen Inglis
Eduardo Martin

Who is the nominated tester for UAT
e.g. John S (don't include surnames in public repos)

Acceptance criteria
To be considered successful the solution must allow:

• The message contains genome size
• The message contains accession number
• The two fields are published correctly in Traction

Dependencies
This story is blocked by the following dependencies:

• #<issue_no.>
• sanger/#<issue_no.>

References
This story has a non-blocking relationship with:

• #<issue_no.>
• sanger/#<issue_no.>

Additional context
Add any other context or screenshots about the feature request here.

User story
As a TOL user I want to create labware in Samples Extraction LIMS using Lab-Share queue system because I want to automatize the transfer of samples and labware information from TOL to PSD.

Who are the primary contacts for this story
Eduardo Martin

Who is the nominated tester for UAT
TOL user (to determine)
Stephen Inglis

Acceptance criteria
To be considered successful the solution must allow:

• Accept a valid lab-share message from TOL coming from the samples reception queue
• Create samples and labware by using the lab-share message information into Samples Extraction LIMS through the reception api
• Send a valid feedback message to the feedback queue so it can be read by TOL apps
• Process with the created labware/samples as normal using Samples Extraction lims
• Design of the solution is documented in lucid charts

Dependencies
This story is blocked by the following dependencies:

• #<issue_no.>
• sanger/#<issue_no.>

References
Lucid chart currently in use:
https://lucid.app/lucidchart/33b22c5e-cd4a-4cd9-8c7c-2ae1f110b347/edit?invitationId=inv_daeccc47-5803-4309-bd63-1a9e8fc4bf6b&page=0_0#

Additional context
A creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware/1-create-labware.avsc

A feedback creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware-feedback/1-create-labware-feedback.avsc

User story
As a TOL user I want to create labware in Traction LIMS using Lab-Share queue system because I want to automatize the transfer of samples and labware information from TOL to PSD.

Who are the primary contacts for this story
Eduardo Martin

Who is the nominated tester for UAT
TOL user (to determine)
Stephen Inglis

Acceptance criteria
To be considered successful the solution must allow:

• Accept a valid lab-share message from TOL coming from the samples reception queue
• Create samples, labware and requests by using the lab-share message information into Traction LIMS through the reception api
• Send a valid feedback message to the feedback queue so it can be read by TOL apps
• Process with the created labware/samples as normal using Traction
• Design of the solution is documented in lucid charts

Dependencies
This story is blocked by the following dependencies:

• #<issue_no.>
• sanger/#<issue_no.>

References
Lucid chart currently in use:
https://lucid.app/lucidchart/33b22c5e-cd4a-4cd9-8c7c-2ae1f110b347/edit?invitationId=inv_daeccc47-5803-4309-bd63-1a9e8fc4bf6b&page=0_0#

Additional context
A creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware/1-create-labware.avsc

A feedback creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware-feedback/1-create-labware-feedback.avsc

User story
As PSD I want to create a demo for the TOL consumer functionality because that will help for testing the decisions for the research story

Who are the primary contacts for this story
Stuart
Eduardo
Andrew

Acceptance criteria
To be considered successful the solution must allow:

• Create a demo that will consume messages and send a feedback: no real process needed, only for testing purposes (maybe we can reuse demo from Heron)
• Use the actual process of validating with Redpanda schemas when reading from Rabbitmq
• Ansible of demo to add queues (CRUD and feedback)
• Ansible of demo to add exchanges and bindings to forward messages to the queues
• Ansible of demo to add users for TOL and PSD
• Add one dead letters exchanges goes on the queues (will forward to the right dead letters queue depending on the response of the consumer (NACK))
• Add 2 dead letter queues (CRUD and feedback deadletters)
• Ansible to add alternate exchange to existing exchanges (publish to this exchange with a routing key that is not bound will forward the message to the alternate exchange)
• Add queues for the alternate exchange (TOL and PSD) depending on the source of message

NOTE

About exchanges: there will be 2 normal exchanges, 2 alternate exchanges and 1 dead letters exchange. There will be 5 exchanges in total.
About queues: there will be 2 normal queues, 2 alternate queues and 2 dead letters queues. There will be 6 queues in total.

The following diagram demonstrates the config for RabbitMQ:

https://lucid.app/lucidchart/4a6ec904-fae1-4e37-90ee-e80d8f32a3b9/edit?invitationId=inv_735c6d91-54ae-46fb-88f3-d57180abeb45

Dependencies
This story is blocked by the following dependencies:

• sanger/General-Backlog-Items#183
• sanger/deployment#250
• sanger/deployment#248

References
This story has a non-blocking relationship with:

• #<issue_no.>
• sanger/#<issue_no.>

Additional context
Add any other context or screenshots about the feature request here.

User story
As TOL user I want to receive feedback from any invalid Samples Extraction lims Lab-share message sent through the queues system because I want to detect any problems on the rejection of the messages sent

Who are the primary contacts for this story
Eduardo Martin

Who is the nominated tester for UAT
Stephen Inglis
TOL user (to determine)

Acceptance criteria
To be considered successful the solution must allow:

• Any semantically incorrect message should sent a rejection feedback message to the feedback queues
• Any message that fails to create labware in Samples Extraction LIMS should send a rejection feedback message with the error produced
• A notification should be received by PSD when a lab-share message is failed (slack message)

Dependencies
This story is related by the following dependencies:

• DPL-546

References
Lucid chart currently in use:
https://lucid.app/lucidchart/33b22c5e-cd4a-4cd9-8c7c-2ae1f110b347/edit?invitationId=inv_daeccc47-5803-4309-bd63-1a9e8fc4bf6b&page=0_0#

Additional context
A creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware/1-create-labware.avsc

A feedback creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware-feedback/1-create-labware-feedback.avsc

Description
As PSD and ToL we need to decide on the message format for the Avro Schema to ensure that the correct information is transferered

Who the primary contacts are for this work
Eduardo
Stuart
Andrew

Acceptance criteria

• Create Avro schema and document describing the CREATE message, with a small set of columns for a sample
• Create Avro schema and document describing the UPDATE message
• Create Avro schema and document describing the FEEDBACK
• Deploy current version of schemas into Redpanda services
• Restrict the value types to be as accurate as possible to the data they represents (eg: use enum, dates, integers)
• The create and update message will have a unique uuid for the message and a uuid for each element you want to have feedback for

Dependencies

• sanger/deployment#250

Knowledge or Stake holders
Other people that may have specific knowledge about this work or have a stake in how it is implemented. e.g. John Smith is an expert on x

Additional context or information
Any other useful context or information that may be useful.

User story
As PSD we need to modify the message schema to allow the required data to be passed from Limber/SS to create the sample/request/library/tube in Traction, for a Pacbio run. Currently it assumes the message is from Tree of Life.

Who are the primary contacts for this story
PSD

Who is the nominated tester for UAT
PSD

Acceptance criteria
To be considered successful the solution must allow:

• consumption of messages from SS/Limber to export a tube into Traction for PacBio

Dependencies
This story is blocked by the following dependencies:

• None

References
This story has a non-blocking relationship with:

• None

Additional context
Add any other context or screenshots about the feature request here.

User story
As TOL user I want to receive feedback from any invalid Traction Lab-share message sent through the queues system because I want to detect any problems on the rejection of the messages sent

Who are the primary contacts for this story
Eduardo Martin

Who is the nominated tester for UAT
Stephen Inglis
TOL user (to determine)

Acceptance criteria
To be considered successful the solution must allow:

• Any semantically incorrect message should sent a rejection feedback message to the feedback queues
• Any message that fails to create labware in Traction LIMS should send a rejection feedback message with the error produced
• A notification should be received by PSD when a lab-share message is failed (slack message)
• A list of errors with error codes must be documented in github/confluence

Dependencies
This story is related by the following dependencies:

• DPL-539

References
Lucid chart currently in use:
https://lucid.app/lucidchart/33b22c5e-cd4a-4cd9-8c7c-2ae1f110b347/edit?invitationId=inv_daeccc47-5803-4309-bd63-1a9e8fc4bf6b&page=0_0#

Additional context
A creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware/1-create-labware.avsc

A feedback creation message will be defined by the schema:
https://github.com/sanger/tol-lab-share/blob/develop/schemas/create-labware-feedback/1-create-labware-feedback.avsc