edinburghgenomics / pyclarity-lims Goto Github PK
View Code? Open in Web Editor NEWThis project forked from scilifelab/genologics
Python interface to the GenoLogics LIMS server via its REST API.
License: MIT License
This project forked from scilifelab/genologics
Python interface to the GenoLogics LIMS server via its REST API.
License: MIT License
At the moment Entity creation can only be done via the Entity contructors such as
Sample.create(...)
It would be useful to have leverage the batch/create function endpoints to create multiple samples/containers.
Function should be called: lims.create_batch
Implementation might be complicated as the entities cannot be created before they are passed on to batch.
The documentation for "Process.all_inputs" infers that the output is always a list type variable but the output variable type can be list or dict_values
Process.all_inputs(self, unique=True, resolve=False) produces a list type variable
Process.all_inputs(self, uniqued=True, resolve=True) produces a dict_values
def all_inputs(self, unique=True, resolve=False):
"""Retrieving all input artifacts from input_output_maps.
If unique is true, no duplicates are returned.
:param unique: boolean specifying if the list of artifacts should be uniqued
:param resolve: boolean specifying if the artifacts entities should be resolved through a batch query.
:return: list of input artifacts._
I can make sure the tests pass, but I don't have access to an instance of Clarity anymore, so don't have any way of validating new versions IRL. Also not sure if it's still in use at Edinburgh Genomics.
@tbooth - it may be good to review this fork versus the upstream and have a think about what should be done with it
add option to open files as binary in lims.get_file_content
For class pyclarity_lims.entities.Process for ""result_files()[source]"
The documentation states:
"Retrieve all resultfiles of output-generation-type perInput."
However, it doesn't seem to do this and instead seems to retrieve all artifacts of output_type "ResultFile:
def result_files(self):
"""Retrieve all resultfiles of output-generation-type perInput."""
artifacts = self.all_outputs(unique=True)
return [a for a in artifacts if a.output_type == 'ResultFile']
It currently accept a protocolStep which is incorrect
Hi all, (Edited from my original post, earlier today, because I was using the wrong data structure for StepPools.pooled_inputs)
Not an issue for you, rather for me! I'm trying to pool 2 plates into 2 tubes. First I create the two pools, set up the step and add the Illumina indexes to the tubes, I think I'm all set here, so you can probably ignore this:
pool1 = Container.create(lims, type=tube_ct)
pool2 = Container.create(lims, type=tube_ct)
pools = [pool1, pool2]
step = Step.create(lims, protocol_step=protocol_step, inputs=artifacts,
container_type_name=tube_ct.name)
process = Process(lims, id=step.actions.step.id)
# Assign the correct Illumina UDI indexes
<snip>
...
sample.artifact.reagent_labels = [label]
sample.artifact.put()
The issue comes when I try and pool the plates. I create a data structure and try and assign this to step.pools.pooled_inputs, thus:
output_pools = {}
for idx, plate in enumerate(plates):
input_arts = []
plate.get(force=True)
out_container = pools[idx]
pool_name = out_container.name
placement_dict = plate.get_placements()
for placekey in placement_dict.keys():
art = placement_dict[placekey]
print 'Artifact:', art
input_arts.append(art)
art_tuple = tuple(input_arts) # tuble of input arts to add to a tube
output_pools[pool_name] = tuple((out_container, art_tuple))
out_container.get(force=True)
for art in input_arts:
art.put()
plate.put()
step.pools.pooled_inputs = output_pools
step.advance()
The error I get is:
step.advance()
File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/entities.py", line 843, in advance
data=self.lims.tostring(ElementTree.ElementTree(self.root))
File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/lims.py", line 172, in post
return self.parse_response(r, accept_status_codes=[200, 201, 202])
File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/lims.py", line 214, in parse_response
self.validate_response(response, accept_status_codes)
File "/usr/local/lib/python2.7/dist-packages/pyclarity_lims/lims.py", line 207, in validate_response
raise requests.exceptions.HTTPError(message)
requests.exceptions.HTTPError: 400: Unpooled inputs exist
I think I've got the data structure correct for StepPools.pooled_inputs. I'm thinking it must be the caching of the artifacts. Can anyone see where I've gone wrong?
Thanks!
Hi,
I am trying to get 'Diluent Volume' out of Clarity LIMs using pyclarity-lims. I can see the UDF if I query the REST API:
<udf:field type="Numeric" name="01_O_QB_Mass">57.84</udf:field>
<udf:field type="Numeric" name="Desired Concentration">2</udf:field>
<udf:field type="Numeric" name="Diluent Volume">2.615</udf:field>
<udf:field type="Numeric" name="Volume to be diluted">1</udf:field>
<udf:field type="Numeric" name="03_I_StockVolume">8</udf:field>
<udf:field type="Numeric" name="02_I_DilutionFactor">7.23</udf:field>
<udf:field type="Numeric" name="01_I_QB_Concentration">7.23</udf:field>
However, I can only return the following UDFs:
{'Desired Concentration': 2.0, '03_I_StockVolume': u'8', 'Volume to be diluted': 1.0, '01_I_QB_Concentration': u'7.23'}
The code I am using looks like this:
for sample in samples:
for outart in process.result_files():
file_sample = outart.samples[0].name
if file_sample == sample.name:
print outart.udf
Anyone see where I am going wrong? The 'Diluent Volume' is set by running a script with trigger_program().
Thanks
Dan
Also modify the Readme to reflect the change of name
Calling the UdfConfig.preset.clear() method seems to clear values from the entire UdfConfig object. Even if I set the preset list manually to a new empty list the object is also cleared...Is this expected?
Which one do we want?
That's definitely too restrictive and causes inconsistencies.
When the UDF does not exist in the dictionary the type is not set which mean no check is performed, setting a string works then.
If the UDF exist then the check is performed and the input value needs to be a date.
Hi,
I'd like to 'Mark protocol as complete' for each sample ONLY IF 'Samples that passed Aggregate_QC...' is true:
I can manually mark as complete using something like:
# TODO: this currently doesn't check the status of the QC flags.
step_actions = step.actions
for artifact in step_actions.next_actions:
actions = artifact['action'] = 'complete'
step.actions.put()
However, I'd like the code to emulate the check that is performed when hitting 'Apply' in the screenshot. Is it just a matter of checking the QC flag for each sample or is there functionality in pyclarity-lims to do this in one easy step as one would do in the web interface?
Thanks
Dan
In the Queue Class I feel as if the "queue-time" member variable should exist under artifacts just like location does. However, I have to traverse artifacts._elems[0]._children[0].text to get the queue time. Can we make this so something like artifacts[0].queue_time works? Thanks.
I'm looking for a way to automate the tracking of reagents using pyclarity-lims. Here I say that a step requires cfDNA_Proteinase_K to be tracked:
However, I cannot see a way to attach a ReagentKit/Lot to the step and therefore I get this error:
Looking at the documentation for step.create I thought I could create a new step with reagent_category=ReagentType.category but I noticed this in the code:
# TODO: more work needed to understand how the permitted reagent applies here
Is the tracking of reagents possible within pyclarity-lims or should I use udfs, or maybe, step.reagent_label?
This is the step that I am trying to automate:
Hi,
I am doing the following but it is very slow:
# Get the the values generated from the 'Calculate DNA Mass' macro
for sample in samples:
for target_file in process.result_files():
target_file.get(force=True) # Does not find it unless force a GET
file_sample = target_file.samples[0].name
if file_sample == sample.name:
print target_file.udf['01_O_TS_Region1Mass']
Is there a different logic I should follow to get the UDFs?
Thanks
Daniel
Hi,
I am getting this warning when using Step.available_programs()
:
/usr/local/lib/python2.7/dist-packages/pyclarity_lims/entities.py:862: FutureWarning: The behavior of this method will change in future versions. Use specific 'len(ele
m)' or 'elem is not None' test instead.
if available_programs_et:
Should I change to a different method to get available programs or is this just related to how the Entities object is going to change in the future?
Thanks!
Dan
Calling lims.get_batch(files)
produces the following error:
File "/usr/lib/python2.6/site-packages/pyclarity_lims/lims.py", line 660, in put_batch
uri = self.get_uri(klass._URI, 'batch/update')
File "/usr/lib/python2.6/site-packages/pyclarity_lims/lims.py", line 78, in get_uri
url = urljoin(self.baseuri, '/'.join(segments))
TypeError: sequence item 2: expected string, NoneType found
It looks like this would be as simple as adding _URI
and _PREFIX
to the File
class.
class File(Entity):
"""File attached to a project or a sample."""
_URI = 'files'
_PREFIX = 'file'
The benefit of supporting this operation is to batch update the is_published
property of files.
update docstring to specify the types of the arguments e.g. inart as string for the artificat id (easily confused with the artifact variable type).
File "./clarity_run_id_setter.py", line 68, in main
Step(lims, id=ep.id).advance()
File "/lustre/home/pipeline/illuminatus/current/_py3_venv/lib/python3.5/site-packages/pyclarity_lims/entities.py", line 843, in advance
data=self.lims.tostring(ElementTree.ElementTree(self.root))
File "/lustre/home/pipeline/illuminatus/current/_py3_venv/lib/python3.5/site-packages/pyclarity_lims/lims.py", line 698, in tostring
self.write(outfile, etree)
File "/lustre/home/pipeline/illuminatus/current/_py3_venv/lib/python3.5/site-packages/pyclarity_lims/lims.py", line 703, in write
etree.write(outfile, encoding='utf-8', xml_declaration=True)
File "/lustre/software/python/Python-3.5.2/lib/python3.5/xml/etree/ElementTree.py", line 772, in write
qnames, namespaces = _namespaces(self._root, default_namespace)
File "/lustre/software/python/Python-3.5.2/lib/python3.5/xml/etree/ElementTree.py", line 875, in _namespaces
for elem in elem.iter():
AttributeError: 'NoneType' object has no attribute 'iter'
Hi,
I got the stack trace above. I'm not sure if this is something broken in the module or just a failure to catch an error condition and convert it to a meaningful error. I think the step was already completed, and I'm not 100% sure I have permission to close it. Will dig further.
Cheers,
TIM
When trying to run integration tests in, e.g, EGCG-Project-Management and trying to import things from within its integration_tests
module, Python instead imports the integration_tests
module from pyclarity-lims because it's installed to site-packages, and then runs into ImportError
s. We should include 'integration_tests' in setup/packages/exclude
I was needing to update the code for another issue and tried on Python 3.9. It doesn't work because of line 648 of lims.py:
for node in root.getchildren():
The documentation for ETree says this should be:
for node in list(root):
As an addendum, I see from issue 58 from last year that this repository may well be dead. Is it time to move to SciLifeLabs original repository?
An empty array is always returned when retrieving a list of entities using the get_*
methods of theLims
class. For example:
samples = l.get_samples(start_index=500)
# samples == []
The problem is in the _get_instances() method. The response is only parsed when start_index
is None
.
I am having an issue when using the API to complete one step and move to the next. I followed the example in https://pyclarity-lims.readthedocs.io/en/latest/PracticalExamples.html#start-and-complete-a-new-step-from-submitted-samples
until the end where I have an error.
When trying to execute the line
actions = s.actions.next_actions[0]['action'] = 'complete'
I get the error of
The valid actions are: [nextstep, repeat, remove, review, rework, completerepeat]
When I try to use the action nextstep
it requires a uri but there isn't documentation of how to assign this. I have tried the line below but that does not work.
actions = s.actions.next_actions[0]['action']['nextsteps'] = 'step_uri_example
Is it possible to create more documentation on how to use this or advise on how to proceed?
I was actually aware of the below issue but it should probably go in the documenatation for pyclarity_lims.entities.Process
- if you want to modify (rather than just view) the UDFs you have to load the step.details
as Dan says.
Hi Tim,
Not sure if you were made aware of this or whether you will ever need
to know. Matt had an issue with one of his scripts and turned out
that it was because he was updating a process UDF using the “process
API end point” and not the “step API end point”. It was also related
to the user role assigned to the person logged in (I know it sounds
strange). So if writing scripts to update step UDFs that may ever be
initiated by a user use the step API end point which in pyclarity-
LIMs speak is something like [PROCESS].step.details.put().Dan
Daniel Forth, PhD
LIMS Manager
We should be able to read utf-16
-encoded files, e.g. from the Spectramax. requests
can return text with any encoding:
>>> request.encoding
'utf-8'
>>> request.text
'some_glitchy_looking_binary_data'
>>> request.encoding = 'utf-16'
>>> request.text
'some_well_formed_data'
We should add this to Lims.get_file_contents
. We should also be able to replace CRLFs with \n
.
When I try to create a queue on version 0.4.2 of this package I am receiving an error. The queue object is created successfully, however when artifacts and queued_artifacts are accessed a stack trace is returned. The stack trace reports "ValueError: unconverted data remains: -0500". I have a feeling this is to due to the queue time for my sample being something like "2018-01-09T15:44:48.571-05:00".
Thank you
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.