aipescience / uws-client Goto Github PK

UWS services can use different namespaces, e.g.
<u:jobs xmlns:u="http://www.ivoa.net/xml/UWS/v1.0">
or
<jobs xmlns:u="http://www.ivoa.net/xml/UWS/v1.0">
instead of
<uws:jobs xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0">
is allowed and should be treated correctly. However, uws-client always expects uws prefix.

toelva:bin adrian$ ./uws -H ... results ...
Downloading result into file 'result.dat'
Traceback (most recent call last):
File "./uws", line 324, in
resultsJob(args.host, args.user, args.pwd, args.id)
File "./uws", line 156, in resultsJob
UWSconn.downloadFile(url, usr, pwd, filename, callback=printProgress)
File "/Users/adrian/Documents/eScience/Tools/uws-client/./lib/UWS/connection.py", line 129, in downloadFile
request.add_header("Authorization", "Basic %s" % self.authStr)
AttributeError: connection instance has no attribute 'authStr'

I'd like to have a flag that suppresses any other output and just returns the id of a newly created job after sending a new job request. I.e. something like:
uws --host https://www.cosmosim.org/uws/query --user <...> --password <...> job new query='...' table='...' --returnjobid
shall return only the jobid, or an error if no job was created.
As it is now, a script using the uws-client would need to parse the output first to find the job id.
Alternatively, we could also return the job parameters as json if a --json flag is set.

Or would you suggest another "hook", maybe importing UWS as a module in a Python script and then use the more basic functions when writing scripts for sending many jobs at once?

When sending requests to a server that returns job lists/details with no uws-prefix, i.e. empty namespace, this fails with the current lxml version (3.5.0) with an error.
Example:
uws --host http://mintaka.ari.uni-heidelberg.de/tap/async/ list
returns in the end:
ValueError: empty namespace prefix is not supported in ElementPath

With lxml version 3.4.4 everything worked fine. Xml with an empty prefix like with this:

<jobs xmlns="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink">`
  <jobref>...</jobref>
  ...

(instead of using xmlns:uws=... and <uws:jobs>, <uws:jobref> etc.) is valid xml and should be allowed. I am going to fix this.

update to UWS 1.1 (or make it optional if STD turns out to be conflicting with UWS 1.0)

UWS1.1. services allow to filter the job list by appending corresponding parameters, copied from the standard, section 2.2.1:

1. Filter by phase, e.g.
   {service-url}/{jobs}?PHASE=EXECUTING

2. {service-url}/{jobs}?AFTER=2014-09-10T10:01:02.000:
   The server should only list jobs with start times after the given [std:iso8601] time in UTC.

3. {service-url}/{jobs}?LAST=100
   The server should list only the given number of most recent jobs ordered by ascending start times.

Filters can be appended, which is to be treated as a union/logical AND.
Servers may respond with a 303 redirect to the jobs-url including a LAST-filter, if the job list is too long.

This would be something really useful for a user, but maybe is better implemented in another script, on top of this generic uws-client?
With CosmoSim's (Daiquiri's) UWS implementation, one can easily filter by the tablename, since it is given in the full job list. For any other parameter, one needs to send not only one request to the job list, but also one request per job to get the details. This can take quite some time and maybe it's a bad idea.
The main problem is that it depends on the UWS-service which parameters are available.
But users do want to search for the ids of jobs with a tablename like 'test-*' or with a service-specific parameter in a specific range or a certain tablename in the sql-query. I would expect a list of jobids returned (together with some details) which can then be used for e.g. deleting all these jobs or looking up further details.

One more thought: at least we could do some filtering for error-messages, since "errorSummary" is included in the uws-schema. And filter for start/endTimes, these are also standardized.

so that refactoring becomes possible

uws [...] job results <jobid> already immediately downloads the results.
What about just showing the result-urls instead (i.e. the results-part of uws job show, mapping it directly to ...../{jobs}/(jobid)/results) and only downloading the results with an additional option like
uws [...] job download <jobid> <resultid>?

This phase is used when the service provider decides to keep the job metadata even after it was deleted or the destruction time of the job has passed.
Jobs in ARCHIVED phase are not to be shown via the job list. They are only available when explicitely demanded using job list filtering: {service-url}/{jobs}?PHASE=ARCHIVED.
See section 2.2.1 in the standard.

UWS models should also be able to generate XML. Then the models can be used as a skeleton to build UWS services in python and this becomes more a library

Does not currently support Python3. First noticed when tried to import client:

ImportError Traceback (most recent call last)
in ()
8 matplotlib.rcParams.update(matplotlib.rcParamsDefault)
9
---> 10 import uws.UWS.client as client
11
12 get_ipython().magic('matplotlib inline')

/usr/local/lib/python3.5/site-packages/uws/UWS/init.py in ()
1 # -- coding: utf-8 --
----> 2 import client
3 import connection
4 import models
5

ImportError: No module named 'client'

The error arises because Python3 needs relative imports to be explicitly defined.

Try e.g. (no credentials needed):
uws --host http://dc.zah.uni-heidelberg.de/tap/async list
and now introduce a typo:
uws --host http://dc.zah.uni-heidelberg.de/taps/async list

The returned error message e.msg is empty, e.raw is just 'False' in handle_error. The response from the server is a "404 Not Found" and should be passed on to the user.