GPL-736 BUG - Small number of duplicated lines in positive reports about lighthouse

[Chris-Friend]

The issue appears to be with the change to batching fetching the cherrypicked samples to add to the positives report. The cherrypicked samples are fetched (through lots of JOINS) using the root sample id. However, the root sample id is not unique to a sample, and fetching on such introduces the possibility of fetching duplicates/fetching the wrong cherrypicked samples.

When there was no batching, there were no duplicate fetches on root sample id, but there was the possibility that the wrong cherrypicked samples were fetched for a positive, as they were not being uniquely identified. This is still the case now, but with batching we can also fetch for the same root sample id twice, meaning we get duplicates.

A fix for both issues would be to uniquely fetch cherrypicked samples using root sample id + other fields

[KatyTaylor]

Deployment and testing:

• Alter connection string variable names in psd-deployment project
• Run report in UAT / training - check for duplicates
• Deploy to UAT / training - inc. deploying new docker config
• Re-run report - check duplicates are gone

linked psd-deployment PR - #160

[KatyTaylor]

Prod report from today 10:30:
Num positives with locations: 574,059
Num positives total: 782,647

Prod report from today 14:11 (after deploy):
Num positives with locations: 574,664 (presume some must have been scanned into LabWhere)
Num positives total: 782,635 (12 fewer, as expected from removing the duplicates)

Searching for the samples that were known duplicates in the report before, I can see that they only appear twice now rather than 4 times.
However, all the samples now say 'No' for the LIMS submission column - will revert the release

[KatyTaylor]

Deployed fix today and results look good, as in above comment. Known 'duplicate' samples now have one row with 'No' and one with 'Yes', as expected if one of them has been cherrypicked and one not.

pre-deploy - 117,146 yes's
post-deploy - 117,128 yes's

implies there were a few that were wrong but not duplicated, same situation I guess but happened to be processed in the same 'chunk'