fgcz / bfabricpy Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 1.0 1.07 MB

API and commad line tools for b-fabric

Home Page: https://fgcz-bfabric.uzh.ch/bfabric/

License: GNU General Public License v3.0

Python 99.23% Makefile 0.08% Shell 0.69%

api bfabric python3 rest-api shiny-r soap wsdl

bfabricpy's People

Stargazers

Watchers

Forkers

cpanse

bfabricpy's Issues

bfabric_save_importresource_sample.py

MaxQuant Status

replacing xpath expression /fastaFiles/FastaFileInfo/fastaFilePath by /srv/www/htdocs/FASTA/fgcz_9606_reviewed_cnl_contaminantNoHumanCont_2016120
9.fasta.
replacing xpath expression /parameterGroups/parameterGroup/fixedModifications/string by Carbamidomethyl (C).
20180918_004_Test_F2_2.raw      //srv/www/htdocs//p2898/Proteomics/QEXACTIVEHF_2/lkunz_20180918_OID11303/20180918_004_Test_F2_2.raw
20180918_003_Test_F2_1_rep.raw  //srv/www/htdocs//p2898/Proteomics/QEXACTIVEHF_2/lkunz_20180918_OID11303/20180918_003_Test_F2_1_rep.raw
20180918_003_Test_F2_1_rep.raw  //srv/www/htdocs//p2898/Proteomics/QEXACTIVEHF_2/lkunz_20180918_OID11303/20180918_003_Test_F2_1_rep.raw
20180918_003_Test_F2_1.raw      //srv/www/htdocs//p2898/Proteomics/QEXACTIVEHF_2/lkunz_20180918_OID11303/20180918_003_Test_F2_1.raw
Configuring 
Testing files 
Feature detection 
bfabric MAXQUANT application failed in '/scratch/MAXQUANT/WU175254/'.

implement test case data for all endpoints for py|java

the following nested data should be ``translated'' to Java, e.g., json | yaml.

self.groundtruth = {'user' : [({'id': 482}, {'login': 'cpanse', 'email':'[email protected]'})],
            'project' : [({'id': 3000}, {'name': 'FGCZ Internal'})],
            'application' : [({'id': 224}, {'name': 'MaxQuant'})],
            'resource' : [({'filechecksum': '090a3f025d3ebbad75213e3d4886e17c'}, {'name': '20190903_07_autoQC4L.raw'}),
                ({'workunitid': 200186}, {'name': '20190618_07_autoQC4L.raw'})],
            'sample' : [({'id': 190249}, {'name': 'autoQC4L'})]
            }

add fastasequence to yaml configuration

bfabricPy/bfabric/bfabric.py

Line 853 in d5a2bc1

if container._classname=="order":

@mariaderrico what do you think?

make a config file reader in YAML

_LOGIN: pfeeder
_PASSWD: dcfxxf742aa459c2abb1095cc247dd49
_WEBBASE: https://fgcz-bfabric.uzh.ch/bfabric
applicationId:
  Metabolomics/Analysis/ProgenesisQI: 226
  Metabolomics/G2HD_1: 81
  Metabolomics/GCT_1: 44
  Metabolomics/IMSTOF_1: 203
  Metabolomics/LTQFT_1: 9
  Metabolomics/ORBI_1: 11
  Metabolomics/ORBI_2: 13
  Metabolomics/ORBI_3: 77
  Metabolomics/QEXACTIVE_3: 171
  Metabolomics/QTOF: 14
  Metabolomics/QUANTIVA_1: 214
  Metabolomics/TOFTOF_2: 143
  Metabolomics/TRIPLETOF_1: 144
  Metabolomics/TSQ_1: 16
  Metabolomics/TSQ_2: 43
  Proteomics/Analysis/GenericZip: 185
  Proteomics/Analysis/MaxQuant: 151
  Proteomics/Analysis/Progenesis: 84
  Proteomics/Analysis/ProteinPilot: 148
  Proteomics/EXTERNAL_0: 188
  Proteomics/EXTERNAL_1: 189
  Proteomics/EXTERNAL_2: 190
  Proteomics/EXTERNAL_3: 191
  Proteomics/EXTERNAL_4: 192
  Proteomics/EXTERNAL_5: 193
  Proteomics/FUSION_1: 162
  Proteomics/FUSION_2: 176
  Proteomics/G2HD_1: 128
  Proteomics/LTQFT_1: 8
  Proteomics/LTQ_1: 7
  Proteomics/LUMOS_1: 248
  Proteomics/ORBI_1: 10
  Proteomics/ORBI_2: 12
  Proteomics/ORBI_3: 87
  Proteomics/PROTEONXPR36: 82
  Proteomics/QEXACTIVEHFX_1: 232
  Proteomics/QEXACTIVEHF_1: 177
  Proteomics/QEXACTIVEHF_2: 197
  Proteomics/QEXACTIVEHF_3: 207
  Proteomics/QEXACTIVE_1: 160
  Proteomics/QEXACTIVE_2: 161
  Proteomics/QEXACTIVE_3: 163
  Proteomics/QTRAP_1: 92
  Proteomics/T100_1: 18
  Proteomics/TOFTOF_2: 91
  Proteomics/TRIPLETOF_1: 93
  Proteomics/TSQ_1: 15
  Proteomics/TSQ_2: 53
  Proteomics/VELOS_1: 90
  Proteomics/VELOS_2: 89

add unit test for mascot feeder

bfabric_flask.py errors when order has no associated samples

I am using bfrabic_flask.py to query b-fabric

When running the R code from bfabricShiny

fromJSON("http://localhost:5000/sample/22342")
Error in open.connection(con, "rb") : HTTP error 500.

The flask server (bfabric_shiny.py) produces this error.
This order has no samples. It might be easier to handle the problem if we get an empty list as a result, instead of an error. Not sure.

Traceback (most recent call last):
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\wewol\AppData\Roaming\Python\Python38\site-packages\flask\app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\wewol\prog\bfabricPy\bfabric\scripts\bfabric_flask.py", line 291, in get_all_sample
    for sample in samples:
TypeError: 'NoneType' object is not iterable

bfabric_save_csv2dataset.py

Hi bfabricPy developers,

I would suggest the following extension of bfabric_save_csv2dataset.py:

a) possibility to incl. additional separator characters during input file parsing

The current implementation strictly requires , as a sep. character. It would be nice if additional ones would be supported (most important tab). I guess that should be easy to do. I would therefor also rename the script to something more generic like bfabric_save_table2dataset.py

b) additional parameter that indicates if the input file contains a header column or not. My impression is that bf ALWAYS expects a header column and the script simply turns the first row into bf attribute names, even in cases where the 1st line already contains values, see https://fgcz-bfabric.uzh.ch/bfabric/dataset/show.html?id=41497&tab=details

Is there a corresponding R wrapper function available? The readme talks about bfabricR, but its not completely clear to me how I would use it.

Best,
Tobi

missing error message exception if login fails

https://github.com/cpanse/bfabricPy/blob/66316d60d522e17686753e26677cdf55f5f3ac01/bfabric/bfabric.py#L70

refactor `urllib.url2pathname`

bfabricPy/bfabric/scripts/bfabric_feeder_mascot.py

Line 198 in 43a16e4

 result=regex0.match(urllib.url2pathname(line.strip()).replace('\\',"/").replace("//","/")) 

make it python3 work

Python 2.7.16 (default, Dec 21 2020, 23:00:36) 
[GCC Apple LLVM 12.0.0 (clang-1200.0.30.4) [+internal-os, ptrauth-isa=sign+stri on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> import sys
>>> import urllib
>>> regex0 = re.compile("^title=.*(p([0-9]+).+Proteomics.*(raw|RAW|wiff)).*")
>>> line="title=File%3a%20%22S%3a%5cp23874%5cProteomics%5cQEXACTIVE_2%5canalytic_20210121%5c20210121_C23874_012_S282311_BC6%2eraw%22%20%3b%20SpectrumID%3a%20%2273256%22%3b%20scans%3a%20%2240490%22"
>>> result=regex0.match(urllib.url2pathname(line.strip()).replace('\\',"/").replace("//","/"))
>>> result
<_sre.SRE_Match object at 0x109ae7ae0>
>>> result.group(1)
'p23874/Proteomics/QEXACTIVE_2/analytic_20210121/20210121_C23874_012_S282311_BC6.raw'
>>> result

Introduce generic paging

bfabricPy/bfabric/bfabric.py

Line 190 in 21963a4

QUERY = dict(login=login, page=page, password=password, query=obj)

pip3 install setup.py is broken

check version numbering in __init__.py

test case for coupling of user triggered applications

also useful as a functional test in

test file

https://github.com/fgcz/bfabricPy/blob/bfabric10/bfabric/tests/test_bfabric_workunit.py

History

https://doi.org/10.1145/1739041.1739135

remove that function

https://github.com/cpanse/bfabricPy/blob/d8cb304b52066c563f5777e2d09185cacc1841d4/bfabric/bfabric.py#L352

howto/make returned bfabric object JSON serializable

https://github.com/cpanse/bfabricPy/blob/ba85e1b2efee9e6dafc33f918dd7c41b08aedf1d/bfabric/scripts/bfabric_read.py#L83

cp@sirius:~ > python3
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bfabric
>>> 
>>> B = bfabric.Bfabric()
--- webbase https://fgcz-bfabric-test.uzh.ch/bfabric; login; cpanse ---
>>> 
>>> user = B.read_object(endpoint = 'user', obj={'login': 'cpanse'})
>>> user
[<suds.sudsobject.xmlUser object at 0x7fdb186087b8>]
>>> import json
>>> json.d
json.decoder          json.detect_encoding( json.dump(            json.dumps(          
>>> json.dumps(user)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type xmlUser is not JSON serializable
>>>

failed: could not get POST content

not sure if this should go to the bfabric ticket system

queue_generator10-www-data-20200921-084731-46489.log

python3

replace suds with zeep
https://github.com/mvantellingen/python-zeep

example on register sample, workunit, resource

add project and order id to yaml created by wrapper_creator

currently the yaml does not contain neither the project_Id or order_Id of the project it is generated for. Some applications e.g. MaxQuant-sampleSizeEstimation would benefit from having this information.

Please also add the workunit Id's of the input resources for the application to the yaml file, If possible.

currently:

application:
  input:
    MaxQuant:
    - [email protected]://home/bfabric/sgeworker/logs//workunitid-242502_resourceid-1673712.out
    - [email protected]://home/bfabric/sgeworker/logs//workunitid-242502_resourceid-1673712.err
    - [email protected]://srv/www/htdocs//p3562/bfabric/Proteomics/MaxQuant/2020/2020-06/2020-06-09/workunit_242502/1673712.zip

TMT sample graph traversal

generate all the output needed for FragPipe TMT run

input: WU yaml config
output:
- sampleGraph.dot
- FragPipe manifest
- annotation

give a meaningfull resource name

https://github.com/cpanse/bfabricPy/blob/f40b0223a0dda4ef6297547ceca2354527d98ee2/bfabric/bfabric.py#L742

refactor bfabric_flask for queue_generator10

https://github.com/cpanse/bfabricPy/blob/1be437e3c0fed059944977d9d5ab1bd3abfd1f1c/bfabric/scripts/bfabric_flask.py#L81

bfabric_save_fasta.py stopped working with bfabric 10

wolski@fgcz-r-028:/scratch/wolski/fasta_db > cat fgcz_7955_danio_rerio_d.txt | bfabric_save_fasta.py 65 /srv/www/htdocs/FASTA/fgcz_7955_danio_rerio_d_20201007.fasta
reading stdin
Traceback (most recent call last):
  File "/usr/local/bin/bfabric_save_fasta.py", line 71, in <module>
    save_fasta(projectid=sys.argv[1], fasta_file=sys.argv[2])
  File "/usr/local/bin/bfabric_save_fasta.py", line 46, in save_fasta
    'applicationid': BFABRICAPPLIATIONID})
  File "/usr/local/lib/python2.7/dist-packages/bfabric/bfabric.py", line 163, in save_object
    return getattr(client.service.save(QUERY), endpoint, None)
  File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 542, in __call__
    return client.invoke(args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/suds/client.py", line 595, in invoke
    soapenv = binding.get_message(self.method, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/suds/bindings/binding.py", line 120, in get_message
    content = self.bodycontent(method, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/suds/bindings/rpc.py", line 56, in bodycontent
    p = self.mkparam(method, pd, value)
  File "/usr/local/lib/python2.7/dist-packages/suds/bindings/binding.py", line 287, in mkparam
    return marshaller.process(content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/core.py", line 62, in process
    self.append(document, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/core.py", line 75, in append
    self.appender.append(parent, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/appender.py", line 102, in append
    appender.append(parent, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/appender.py", line 243, in append
    Appender.append(self, child, cont)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/appender.py", line 182, in append
    self.marshaller.append(parent, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/core.py", line 75, in append
    self.appender.append(parent, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/appender.py", line 102, in append
    appender.append(parent, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/appender.py", line 243, in append
    Appender.append(self, child, cont)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/appender.py", line 182, in append
    self.marshaller.append(parent, content)
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/core.py", line 74, in append
    if self.start(content):
  File "/usr/local/lib/python2.7/dist-packages/suds/mx/literal.py", line 87, in start
    raise TypeNotFound(content.tag)

enable caching

https://github.com/cpanse/bfabricPy/blob/ade32c17efe004b2e520360d35508cd5198a154d/bfabric/bfabric.py#L164

extent sample test case with `Library - illumina`

https://github.com/cpanse/bfabricPy/blob/086dade16ffa3382cbafa29a905d11bf0b5e0270/bfabric/tests/test_bfabric_sample.py#L21

implement:

def sample_save_library_illumina(self):
    pass

allow empty parameter

https://github.com/cpanse/bfabricPy/blob/5054bfc6950a9025926c7d466d19d80684421acc/bfabric/bfabric.py#L697

reconsider to turn the repository from private to public

refactor using f-string

'app.json_encoder' is deprecated

bfabricPy/bfabric/scripts/bfabric_flask.py

Line 94 in d5be077

app.json_encoder = BfabricJSONEncoder

/home/cpanse/__checkouts/bfabricPy/bfabric/scripts/./bfabric_flask.py:94: DeprecationWarning: 'app.json_encoder' is deprecated and will be removed in Flask 2.3. Customize 'app.json_provider_class' or 'app.json' instead.

make intro videos

bash

python

R

change email || make email config

e.g.,

[email protected]

bfabricPy/bfabric/bfabric.py

Line 633 in 9a41cad

export EMAIL="[email protected] [email protected] [email protected]"

to receive an email you have to register in the ticket system

workunit is always set to status available

independent if the job failed.

could be caused by the sleep 10

make `python3 -m unittest fgcz_maxquant_wrapper.py` pass

cp@dhcp-wlan-eduroam-10-16-0-099:~/__checkouts/bfabricPy/bfabric/scripts  (bfabric9)> python3 -m unittest fgcz_maxquant_wrapper.py 
.../Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/yaml/constructor.py:126: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if not isinstance(key, collections.Hashable):
no scratch dir '/scratch/MAXQUANT/WU181492'.
E
======================================================================
ERROR: test_xml (fgcz_maxquant_wrapper.TestFgczMaxQuantConfig)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/cp/__checkouts/bfabricPy/bfabric/scripts/fgcz_maxquant_wrapper.py", line 670, in test_xml
    MQC.generate_mqpar("/tmp/output.xml", xml_template=mqpartree)
  File "/Users/cp/__checkouts/bfabricPy/bfabric/scripts/fgcz_maxquant_wrapper.py", line 89, in generate_mqpar
    element = xml_template.find(query)
  File "src/lxml/etree.pyx", line 1527, in lxml.etree._Element.find
  File "src/lxml/_elementpath.py", line 311, in lxml._elementpath.find
  File "src/lxml/_elementpath.py", line 300, in lxml._elementpath.iterfind
  File "src/lxml/_elementpath.py", line 269, in lxml._elementpath._build_path_iterator
  File "<string>", line None
SyntaxError: cannot use absolute path on element

----------------------------------------------------------------------
Ran 4 tests in 0.030s

FAILED (errors=1)

Getting number of pages with BFabricPy

Hi team,

Is there a way to get the total number of pages using the BFabricPy python API? It seems the most recent release makes it so a suds xmlResponse is returned with the requested page and number of pages, while in the past this simply returned None. I am wondering if it is possible to know this information a priori.

Thanks!
Falko

Support for python 3.9

Hi there dev team

Would it be possible to add support for python 3.9? I have it working just changing the dependency requirements in the setup files, but this is obviously a bit risky in the long-run.

Much appreciated.

change submit dir from `/tmp` to `/home/bfabric/prx`

bfabricPy/bfabric/bfabric.py

Line 648 in 5d33ba9

_bash_script_filename = "/tmp/externaljobid-{0}_executableid-{1}.bash"\

Serialization of returned bfabric objects to yaml format

Considering the option of generalizing the serialization to either json or yaml format, I have tested the yaml.dump command on the bfabric output application but I couldn't print the whole content in a clean manner, and some of the values are replaced by i.e. !!python/object/new:suds.sax.text.Text (see example of output below).

I have explored the available parameters of the yaml.dump function but no luck so far.
Yaml equivalents seem to be missing for some of the suds objects, which might require to implement specific mappings if not solved otherwise. Any help would be welcome!

import yaml
import bfabric

B = bfabric.Bfabric()
application = B.read_object(endpoint = 'application', obj={'id': '61'})
print(yaml.dump(application))

Some of the lines as printed by the command:

    [....]
    - _id
    - _classname
    __metadata__: !!python/object:suds.sudsobject.Metadata
      __keylist__:
      - sxtype
      __printer__: !!python/object:suds.sudsobject.Printer {}
      sxtype: *id1021
    __printer__: !!python/object:suds.sudsobject.Printer {}
    _classname: !!python/object/new:suds.sax.text.Text
      args:
      - user
      state:
        escaped: false
        lang: null
    _id: 482
  technology: !!python/object/new:suds.sax.text.Text
    args:
    - Proteomics
    state:
      escaped: false
      lang: null
  type: !!python/object/new:suds.sax.text.Text
    args:
    - Import
    state:
      escaped: false
      lang: null
  valid: !!python/object/new:suds.sax.text.Text
    args:
    - 'true'
    state:
      escaped: false
      lang: null

add createby in wrappercreator config

bfabric10 -> empty parameter in yaml configuration

bfabricPy/bfabric/bfabric.py

Line 784 in d5ba685

 parameter = self.read_object('parameter', obj={'id': parameter._id, 'context': 'APPLICATION'}) 

cp@sirius:~/__checkouts/bfabricPy/bfabric  (bfabric10)> bfabric_read.py parameter context APPLICATION
--- webbase https://fgcz-bfabric.uzh.ch/bfabric; login; pfeeder ---
bfabricPy version 0.10.16 (2020-06-15) -- "suds-py3"    
Copyright (C) 2019 Functional Genomics Center Zurich

--- query = {'context': 'APPLICATION'} ---
--- Empty result set or invalid query. ---
cp@sirius:~/__checkouts/bfabricPy/bfabric  (bfabric10)>

barcode endpoint

Hi team,

Would it be possible to add "barcode" to the list of supported bfabric endpoints?

Thank you!

Time-out error when using bfabric.save_object for large number of objects.

Hi bfabricPy-Team,

I am encountering a time-out error when I try to update a large amount of samples using bfabricPy. For this particular case, I am passing 485 objects to bfabricPy in total. As the below error implies, writing this information appears to work without issue (confirmed when looking at the updated information in BFabric), but when it comes time to read this information back, it appears some time-out is reached and the process errors out. I will rework my code to split the operation into multiple calls, but I figured I'd at least let you know.

2023-03-19 17:53:27,622 - ERROR❌ : [draugr:250] The read operation timed out
Traceback (most recent call last):
  File "/export/local/analyses/draugr_exec/draugr.py", line 228, in main
    post_demuxer.post_demux(update_sample_barcodes=args.no_update_bf_barcodes)
  File "/misc/ngseq10/opt/seqtools/draugr/src/post_demultiplex.py", line 55, in post_demux
    self._update_sample_barcodes()
  File "/misc/ngseq10/opt/seqtools/draugr/src/post_demultiplex.py", line 268, in _update_sample_barcodes
    res = suds_to_json(self._bfc.write_obj("sample", sample_barcodes))
  File "/misc/ngseq10/opt/seqtools/draugr/src/bfabric/client.py", line 88, in write_obj
    return self.bfabric.save_object(endpoint, obj)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/bfabric/bfabric.py", line 228, in save_object
    return getattr(self.cl[endpoint].service.save(QUERY), endpoint, None)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/suds/client.py", line 559, in __call__
    return client.invoke(args, kwargs)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/suds/client.py", line 618, in invoke
    result = self.send(soapenv)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/suds/client.py", line 652, in send
    reply = transport.send(request)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/suds/transport/http.py", line 178, in send
    return HttpTransport.send(self, request)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/suds/transport/http.py", line 78, in send
    fp = self.u2open(u2request)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/site-packages/suds/transport/http.py", line 119, in u2open
    return url.open(u2request, timeout=tm)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/urllib/request.py", line 1350, in do_open
    r = h.getresponse()
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/http/client.py", line 1371, in getresponse
    response.begin()
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/http/client.py", line 319, in begin
    version, status, reason = self._read_status()
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/http/client.py", line 280, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/ngseq/miniconda3/envs/gi_py3.9.7/lib/python3.9/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

All the best,
Falko

assign application executable to externaljob

(hard constrain test by bfabric)

and check whenever we create an externaljob!!

bfabricPy/bfabric/bfabric.py

Line 842 in 5eeda3b

{"workunitid": workunit._id,

iff a `dataset` exists include it in the yaml file generated by the wrapper creator

https://github.com/cpanse/bfabricPy/blob/ba85e1b2efee9e6dafc33f918dd7c41b08aedf1d/bfabric/bfabric.py#L549

the following code snippet is part of the solution (see fgcz fragpipe project)

def extract_design_matrix(ds, outputfile=None):
    required_columns = {'Resource': [], 'Relative Path': [], 'Experiment': [], 'Replicate': []}

    position = -1
    for c in required_columns.keys():
        for a in ds.attribute:
            if a.name == c:
                postition = a.position
                continue

        for i in ds.item:
            t = map(lambda x: (x.attributeposition, x.value), i.field)
            t = filter(lambda x: x[0] == postition, t)

            if c in required_columns:
                value = "{}".format(list(t)[0][1])
                required_columns[c].append(value)

    with open(outputfile, "w") as f:
        for (k, v) in required_columns.items():
            f.write("{}={}\n".format(k, "\t".join(v)))

query_obj = {'id': wu.inputdataset._id}
ds = bfapp.read_object(endpoint='dataset', obj=query_obj)[0]

Type not found: xmlRequestSaveParametersWorkunit by creating multiple clients of the same endpoint

https://github.com/cpanse/bfabricPy/blob/e0ff9c2a70b9744757e1c2bd5e51d304f96e1b6c/bfabric/bfabric.py#L173

if I create only one client it works.

#!/usr/bin/env python3
# -*- coding: latin1 -*-

import sys

try:
    from suds.client import Client
    from suds.client import WebFault
except:
    raise

from pprint import pprint

import hashlib
import os
import base64
import datetime
import re
import unittest

if (sys.version_info > (3, 0)):
    import http.client
    http.client.HTTPConnection._http_vsn = 10
    http.client.HTTPConnection._http_vsn_str = 'HTTP/1.0'
    pass
else:
    import httplib
    httplib.HTTPConnection._http_vsn = 10
    httplib.HTTPConnection._http_vsn_str = 'HTTP/1.0'


if __name__ == "__main__":
    _WEBBASE="https://fgcz-bfabric.uzh.ch/bfabric"
    _LOGIN="cpanse"
    _PASSWD="XXXXXXXXXXXXXXXXXXX"

    # create a client for every endpoint
    endpoints = ['annotation', 'application',
        'comment', 'executable',
        'externaljob', 'importresource', 'mail',
        'parameter', 'project', 'resource', 'sample',
        'storage', 'user', 'workunit', 'order']
    cl = dict()
    for endpoint in endpoints:
        try:
            cl[endpoint] = Client("".join((_WEBBASE, '/', endpoint, "?wsdl")), cache=None)
        except:
            print (endpoint)
            raise

    obj = {'login': _LOGIN, 'page': '', 'password':_PASSWD, 'query':{}}
    for i in range(1, 10):
        res = cl['storage'].service.read(obj)
        print (res)

    obj = {'login': _LOGIN, 'page': '', 'password':_PASSWD, 'query':{'id': 123456}}

    for i in range(1, 10):
        res = cl['workunit'].service.read(obj)
        print (res)

add try/except block

https://github.com/cpanse/bfabricPy/blob/c388b7eedfa007f9a9be1e1a062a8fd25519bb28/bfabric/bfabric.py#L711

Implementing unittest in test_bfabric_read.py module

In order to perform unittest on the script bfabric_read.py, the module test_bfabric_read.py has to:

call bfabric_read.main(endpoint, query) (insted of res = self.bfapp.read_object(endpoint=endpoint, obj=query))
compare results with the groundtruth by capturing the bfabric_read output from stdout

To do so the following modifications become necessary:

in bfabric_read.py move the main code into a main(endpoint, query) function
make the bfabric_read.main accessible from test_bfabric_read.py:
- create bfabric/scripts/__init__.py
- add bfabric.scripts package to setup.py: packages = ['bfabric','bfabric.scripts']

read_object query print statement

Hi team,

There appears to be a print statement in the read_object function in the newest version which can not be toggled on or off:
https://github.com/fgcz/bfabricPy/blame/bfabric12/bfabric/bfabric.py#L209

It would be nice if this would be off by default or at least toggable to off with verbose=False or something similar, as the current state of the code clutters up a lot of my own logging.

Would that be possible?

All the best,
Falko

bfabric wsdl resource id 550327 using python3 pkg zeep - Unexpected element 'created'

I experience problems when I query for the resource's endpoint using the python3 zeep (replaces suds) package.

output of a query of all orders within service areas: 28, 5 and 14 as a text file

Hi Christian,

Whenever you have time (not urgent) do you think you could send me the output of a query of all orders within service areas: 28, 5 and 14 as a text file from the productive system? Or the script you would run to generate such a query? I didn't yet figure out how to query bfabric9.

Would be really helpful! Thank you

Best,

Griffin

py2 -> py3 make bf object global - pass login and webpassword as agrument

https://github.com/cpanse/bfabricPy/blob/ade32c17efe004b2e520360d35508cd5198a154d/bfabric/scripts/bfabric_flask.py#L106

@app.route('/q', methods=['GET', 'POST'])
def q():
    try:
        content = json.loads(request.data)
    except:
        return jsonify({'error': 'could not get POST content.'})

    try:
        webservicepassword = content['webservicepassword'][0].replace("\t", "")
        login = content['login'][0]

        # print("endpoint={}\n\tquery={}\n".format(content['endpoint'][0]), content['query'])
        print(content['endpoint'][0])
        print(content['query'])

        res = bfgapp.read_object(endpoint=content['endpoint'][0], obj=content['query'], login=login, password=webservicepassword)

        logger.info("'{}' login success query {} ...".format(login, content['query']))
    except:
        logger.info("'{}' login failed ...".format(login))
        return jsonify({'status': 'jsonify failed: bfabric python module.'})

    try:
        return jsonify({'res': res})
    except:
        logger.info("'{}' query failed ...".format(login))
        return jsonify({'status': 'jsonify failed'})

fgcz / bfabricpy Goto Github PK

bfabricpy's People

Stargazers

Watchers

Forkers

bfabricpy's Issues

test file

History

bash

python

R

Recommend Projects

Recommend Topics

Recommend Org

Jobs