Comments (9)
Meeting with Rob A (PAM) on 8th June to agree
What information they will send
What changes in anything to the export
What format
from lighthouse.
Tasks:
-
Define the api message in github with success (200) and error situations descriptions with examples
value_in_sequencing allowed values = ['Yes', 'No', 'Unknown']
declared_at allowed format = r"%Y-%m-%dT%H:%M:%S" -
Allow creation of the entries in samples_declarations table.
POST request accepts the following data, a list of samples
[{"root_sample_id": "0006","value_in_sequencing": "Yes", "declared_at": "2013-04-04T10:29:13"}, {"root_sample_id": "0007","value_in_sequencing": "Unknown", "declared_at": "2013-04-04T10:29:13"}]
-
validate sample exists in samples table
-
validate value_in_sequencing is a either ["Yes", "No", "Unknown"]
-
validate declared_at is a date
-
Test performance of sending/receiving 100000 samples with the new api blueprint. Check size of data because the request is going to be 100.000 samples (5MB)
-
Create new table with name
sample_description
, to store{ valid_in_sequencing, sample_id, updated_at }
-
Add a new validation to check that the samples description has unique samples ids inside the bulk_insert
-
Add to the Eve middleware a check for validation API key same as in Aker
generate and send secret key -
build join query:
join samples table andsample_description
table
where sample id is the sample -
modify jobs report to use this query, to generate the excel file
(blueprints/reports.py get_reports_details) -
When creating the plates in SS, (blueprints/plates.py)
send the setting for value_in_sequencing to be stored in sample_metadata table.
(CHECK: required in SS?)
In SS, add value_in_sequencing field to sample_metadata table,
add to SS api to insert data into table
add to msg to MLWH
add value_in_sequencing field to mlwh table
Nice to have, but not important
Check in 4 if is possible to merge from 2 different tables to answer the request
- Update the samples api in lighthouse to include this value_in_sequencing
from lighthouse.
This issue belongs in the lighthouse service.
from lighthouse.
Update on acceptance criteria (from mail chain 9/Jun/2020)
- LIMS service receives a request to http/s api endpoint from a server inside the Sanger intranet
- The request has a secret key as header
- Service needs to support:
- At initial stages, uploading the update of 50.000 - 100.000 samples once/twice every week
- In future, receiving a delta (diff of samples) with 100-1000 once/twice a week
- All samples will be assigned a time and date (value_in_sequencing_updated_at) that will correspond with the time and date when we received this request in LIMS.
- The message of the request will contain a list of samples information with:
- Root Sample ID
- Value in sequencing (Y/N)
- Date when value in sequencing is declared
- If we receive the same sample id again, we will update its Y/N value in value_in_sequencing and the date in value_in_sequencing_updated_at.
- In case of success, the answer will be the same list of samples with the updated values. It will return a HTTP success code (200).
- In case of error it will return a HTTP error (5xx, 4xx) depending in the condition.
- The error conditions in message:
- Root sample id not present (5xx)
- Incorrect message format (4xx)
- In case of error, apply the message partially with all right samples and receive back a list of successful and error updates
- The automatic report of positive samples on site will be generated with current settings of value_in_sequencing and value_in_sequencing_updated_at
from lighthouse.
API documentation: https://ssg-confluence.internal.sanger.ac.uk/display/PSDPUB/LightHouse+Samples+declarations+API+Documentation
from lighthouse.
# curl -X POST http://127.0.0.1:5000/samples_declarations -d '[{"root_sample_id": "0012", "declared_at": "2013-04-04T10:29:13"}, {"root_sample_id": "0012", "declared_at": "2013-04-04T10:29:13"}, {"root_sample_id": "0010", "declared_at": "2013-04-04T10:29:13"}, {"root_sample_id": "0011", "declared_at": "2013-04-04T10:29:13"}, {"root_sample_id": "0011", "declared_at": "2013-04-04T10:29:13"}]' -H 'Content-Type: application/json'
from lighthouse.
Performance
10000 samples = 14s
20000 samples = 30s
30000 samples = 69s
40000 samples = 102.94s
from lighthouse.
Requested update from PAM Friday, 26 June 2020 at 09:44
from lighthouse.
Christoph P writes Friday, 26 June 2020 at 12:53
Hello again,
have done some testing now. Can confirm that the service is working well, many thanks! Error messages are also very clear, which is helpful. Couple of things to make you aware of - none of these is critical, but you might want to look into them:
when providing a JSON with errors, I do not get the full result (with fields "_updated", "_created" etc) for the records that are okay, so unclear whether they were registered or not (I think we agreed to accept the correct records and just fail the incorrect ones, see further down in the thread). Would be good to get clarity of what is happening.
when providing a JSON with a missing sample ID, I do not get a JSON with error message, but an html with "internal server error"
when re-uploading sample IDs, I would expect the "_updated" and "_created" to differ in the response, but they are always the same and show the most recent submission
Cheers,
Christoph
from lighthouse.
Related Issues (20)
- DPL-470 - LSPA date tested in future HOT 1
- DPL-472 - Brants Bridge sent root sample ids in incorrect format
- DPL-473 - Brants Bridge sent ~1500 samples with incorrect RNA ID HOT 1
- DPL-342 Remove report [C=S, V=3]
- DPL-362 As GSU (Alan K) I want to allocate a unique COG-UK ID to all historic Heron samples so that we can use this instead of Root Sample ID within our internal tracking. HOT 10
- DPL-426 Remove updates for MLWH once all samples arrive via RabbitMQ HOT 2
- DPL-429 As developer I want to remove the lighthouse reports generation as is not in use anymore and want to have less code to maintain HOT 1
- DPL-544 Make data consistent between MongoDB, MLWH and Sequencescape
- DPL-614 Missing plate map (Heron)
- DPL-631 [BUG] Incorrect evaluation of must_sequence affects both Box buster and Biosero HOT 1
- DPL-629 Check status of Heron CP plates HOT 1
- DPL-675-2: AROUND SEPTEMBER 2023 -- Remove reports and Sentinel functionality HOT 1
- DPL-717-1: Create a new endpoint for creating entities for Lighthouse deep-well plates in Sequencescape
- DPL-757 As PSD I want to create a first draft design for RVI new project (multipick) that will reflect lims architecture and required components to solve the current list of user requirements obtained.(S=M; C=L) HOT 1
- DPL-776 Investigate a way to identify Heron/RVI samples already in SequenceScape so that we can avoid problems during stamping from deep wells to shallow well plates for RVI.
- DPL-782 Filter samples being imported for deep-well to shallow-well stamping so as to exclude samples already in Sequencescape
- DPL-821 [RVI] Cherry-picking LSW-96 Stock plates using the existing Tecan systems HOT 4
- DPL-670-4 Pass a Lighthouse specific API key to Sequencescape calls HOT 3
- DPL-840 Populate missing sample fields in MLWH and consequently in CoTrack
- Y24-191 - Use the v2 API key for all Sequencescape requests
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lighthouse.