icrar / ngas Goto Github PK

NGAS has a feature through which, at startup, a server can create (and later destroy, at shutdown) subscription from external NGAS servers. This feature is governed by the SubscriptionDef XML element of the configuration file, which allows multiple subscription specifications (i.e., the servers where data will be received from) to be given via their respective Subscription XML sub-elements.

When a server A processes a Subscription element to create a subscription for data from B, B must be given a URL that points to A so that it can successfully push data. This URL was historically specified using the SubscriberUrl attribute of the Subscription element. In 1fd12a0 this changed though and a new, simpler Command attribute was introduced. The attribute should only contain the command to invoke on A, while the full URL would be automatically calculated by the code. The reasoning for this change was:

Users should not need to specify the full URL in the configuration file, because that will depend on the IpAddress that is set in the Server element (and in the NAT/reverse proxy information that we need to add to the server as well).

In other words, specifying A's own URL on each of its Subscription elements was deemed as duplicate, and potentially conflicting, information.

In principle this makes sense, but it has presented at least two different problems:

• The current implementation breaks when the IpAddress element is set to 0.0.0.0. In such cases we can only guess what the correct IP should be used to construct the URL.
• The commit quoted above talked about handling NAT/reverse proxy information, but this has never been implemented.

Of these two, the first problem is more urgent than the second. And in both cases the problem can be solved by reinstating SubscriberUrl as the main mechanism to feed A's URL into B, even if that means duplicating some information. We will thus perform this change, but still letting Command be specified and used if the former is not specified.

This issue was originally pointed out by @smclay.

This is related to #8 (I'd be building this on top of #8).

Currently, the subscription service client can only use passwords. We (DC) are planning on using JSON Web Tokens, so we need some kind of framework to add new authentication methods. I plan on having a new section in the config file, which allows specifying a module which provides an object (probably called auth) which handles the authentication part (and which the subscription service can be told to use). I'll write up a PR which does this (which will need #8).

Hello,

I have a "working" NGAS server running but I am not able to push even a basic FITS file without error.

Code snippet (python):

import requests
with open('test.fits', 'rb') as f: 
    url = 'http://localhost:7778/QARCHIVE?filename=test.fits' 
    r = requests.post(url, data=f)

Result (server-side):

KeyError: "Keyword 'ARCFILE' not found."

This seems to be originating in the getDpIdInfo code in ngamsFitsPlugIn.py. I'm a little confused why this code should be running, as I have not read that an 'ARCFILE' key in the FITS header is required for NGAS and the NGAS filename (I thought) is specified in the http request.

Is this expected default behaviour?

P.S. I tried to upload the FITS file but apparently it is not supported by GitHub... Let me know if you require it. It was generated along with its default header from scratch using astropy (I have also tried other FITS files, with the same result).

During the discussions of #28, @smclay pointed out that the implementation of the proxy_request method in the ngamsServer reads all the incoming response data before sending it back to the original client. This is fine for small amounts of data, but adds latency to the response, and more importantly has the potential of crashing NGAS if the memory limits of the machine are reached.

This issue is to re-implement proxy_request so it streams the incoming data instead of accumulating it in memory. Once this is implemented we could take care of merging the functionality currently being discussed in #28 to avoid some potential code duplication.

either use six.moves.input() or at the top of the file: from six.moves import input and change line#172 accordingly

--- a/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
+++ b/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
@@ -169,10 +169,10 @@ def console_input(message):
     :param message: Message to print (string)
     :return: Information entered by the user (string)
     """
-    return six.input("INPUT> " + message + " ").strip()
+    return six.moves.input("INPUT> " + message + " ").strip()

(Found by Nathalie Fourniol as a runtime error and verified with mypy)

After I added proper type annotations to ngamsUtils/ngamsUtils/ngasUtilsLib.py, function decrypt_access_code()
and changed the deprecated base64.decodestring() to base64.decodebytes():

diff --git a/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py b/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
index 41ea97ee..d19a1b2c 100644
--- a/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
+++ b/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
@@ -130,14 +130,14 @@ def encrypt_access_code(access_code):
     return base64.encodestring(access_code)


-def decrypt_access_code(encrypted_access_code):
+def decrypt_access_code(encrypted_access_code: str) -> str:
     """
     Decode an Access Code as used by the NGAS Utilities

     :param encrypted_access_code: Encrypted Access Code (string)
     :return: Decoded Access Code (string)
     """
-    return base64.decodestring(encrypted_access_code)
+    return base64.decodebytes(encrypted_access_code)

Then when I run mypy ngamsUtils/ngamsUtils/ngasUtilsLib.py I get:

ngasUtilsLib.py:140: error: Incompatible return value type (got "bytes", expected "str")
ngasUtilsLib.py:140: error: Argument 1 to "decodebytes" has incompatible type "str"; expected "Union[bytes, Union[bytearray, memoryview, array[Any], mmap, _CData, PickleBuffer]]"

The fix for this is (feel free to remove type annotations):

diff --git a/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py b/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
index 41ea97ee..aaf1c8f4 100644
--- a/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
+++ b/src/ngamsUtils/ngamsUtils/ngasUtilsLib.py
@@ -130,14 +130,14 @@ def encrypt_access_code(access_code):
     return base64.encodestring(access_code)


-def decrypt_access_code(encrypted_access_code):
+def decrypt_access_code(encrypted_access_code: str) -> str:
     """
     Decode an Access Code as used by the NGAS Utilities

     :param encrypted_access_code: Encrypted Access Code (string)
     :return: Decoded Access Code (string)
     """
-    return base64.decodestring(encrypted_access_code)
+    return base64.decodebytes(encrypted_access_code.encode()).decode('utf-8')

Hi,

I'm using ngas v11.0.2 tag and following https://ngas.readthedocs.io/en/latest/install.html#total-system-setup

This is on a clean install of ubuntu18.04 server, with a virtual env built with:
create_venv.sh -f -2 -p /usr/bin/python2.7 /home/mwa/ngas_rt

Have you seen this before? Any ideas on what dependency I am missing? I can't find anything similar on google to help either!

Here's the full output:
(ngas_rt) mwa@mwacache01:~/ngas_src$ pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
asn1crypto==1.0.1
bcrypt==3.1.7
boto==2.49.0
cffi==1.12.3
cryptography==2.7
enum34==1.1.6
fabric==2.5.0
invoke==1.3.0
ipaddress==1.0.22
paramiko==2.6.0
pycparser==2.19
pycrypto==2.6.1
PyNaCl==1.3.0
six==1.12.0

(ngas_rt) mwa@mwacache01:~/ngas_src$ fab hl.operations_deploy -u gsleap --set NGAS_USER=mwa
Traceback (most recent call last):
File "/home/mwa/ngas_rt/bin/fab", line 10, in
sys.exit(program.run())
File "/home/mwa/ngas_rt/local/lib/python2.7/site-packages/invoke/program.py", line 373, in run
self.parse_collection()
File "/home/mwa/ngas_rt/local/lib/python2.7/site-packages/invoke/program.py", line 465, in parse_collection
self.load_collection()
File "/home/mwa/ngas_rt/local/lib/python2.7/site-packages/fabric/main.py", line 87, in load_collection
super(Fab, self).load_collection()
File "/home/mwa/ngas_rt/local/lib/python2.7/site-packages/invoke/program.py", line 696, in load_collection
module, parent = loader.load(coll_name)
File "/home/mwa/ngas_rt/local/lib/python2.7/site-packages/invoke/loader.py", line 76, in load
module = imp.load_module(name, fd, path, desc)
File "/home/mwa/ngas_src/fabfile/init.py", line 30, in
from . import aws
File "/home/mwa/ngas_src/fabfile/aws.py", line 29, in
from fabric.colors import green, red, blue, yellow
ImportError: No module named colors

Function base64.b64decode() returns bytes in Python3 whereas variable password is supposed to be a string. An explicit conversion is missing:

-    password = base64.b64decode(password)
+    password = base64.b64decode(password).decode('utf-8')

When the ngams server is running in our configuration ps shows three running processes, corresponding (I think) to main, data-check and janitor.
If we send an INIT command, we end up with 2 running processes and one defunct process. We suspect the defunct process is the Janitor task.
The same issue happens when we send OFFLINE followed by ONLINE.

It would appear that when NGAS receives a request for a new subscription it does not check if the subsriber ID is already in use before attempting to create a new entry in the NGAS_SUBSCRIBERS table...

2021-04-21T16:34:41.083 [ 1221] [ R-1758806] [  INFO] ngamsServer.ngamsServer#handleHttpRequest:1806 Handling HTTP request: client_address=('134.171.18.32', 32850) - method=GET - path=|SUBSCRIBE?subscr_id=aat-ngas-6%3A8002&priority=1&url=http%3A%2F%2Faat-ngas-6.hq.eso.org%3A8002%2FQARCHIVE&start_date=2021-04-21T18%3A34%3A41.078|
2021-04-21T16:34:41.084 [ 1221] [ R-1758806] [  INFO] ngamsServer.ngamsCmdHandling#_get_module:74 Received command: SUBSCRIBE
2021-04-21T16:34:41.084 [ 1221] [ R-1758806] [  INFO] ngamsServer.commands.subscribe#handleCmd:110 Creating subscription for files >= 2021-04-21T16:34:41.078
2021-04-21T16:34:41.093 [ 1221] [ R-1758806] [ ERROR] ngamsServer.ngamsServer#reqCallBack:1743 Error while serving request
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsServer.py", line 1727, in reqCallBack
    method, path, headers)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsServer.py", line 1822, in handleHttpRequest
    ngamsCmdHandling.handle_cmd(self, reqPropsObj, httpRef)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsCmdHandling.py", line 63, in handle_cmd
    msg = _get_module(srvObj, reqPropsObj).handleCmd(srvObj, reqPropsObj, httpRef)
  File "/usr/lib/python2.7/site-packages/ngamsServer/commands/subscribe.py", line 133, in handleCmd
    addSubscriber(srvObj, subscrObj)
  File "/usr/lib/python2.7/site-packages/ngamsServer/commands/subscribe.py", line 55, in addSubscriber
    srvObj.getDb().insertSubscriberEntry(subscrObj)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsDbNgasSubscribers.py", line 156, in insertSubscriberEntry
    self.query2(sql, args = vals)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsDbCore.py", line 837, in query2
    return t.execute(sqlQuery, args)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsDbCore.py", line 837, in query2
    return t.execute(sqlQuery, args)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsDbCore.py", line 572, in execute
    cursor.execute(sql, args)
  File "/usr/lib/python2.7/site-packages/DBUtils/SteadyDB.py", line 608, in tough_method
    result = method(*args, **kwargs)  # try to execute
IntegrityError: ORA-00001: unique constraint (NGAS05_AAT.IDX_NGAS_SUBSCRIBERS_SUBSCR_ID) violated
2021-04-21T16:34:41.094 [ 1221] [ R-1758806] [  INFO] ngamsServer.ngamsServer#send_status:350 Returning status FAILURE with message ORA-00001: unique constraint (NGAS05_AAT.IDX_NGAS_SUBSCRIBERS_SUBSCR_ID) violated and HTTP code 400
2021-04-21T16:34:41.094 [ 1221] [ R-1758806] [  INFO] ngamsServer.ngamsServer#send_status:350 Returning status FAILURE with message ORA-00001: unique constraint (NGAS05_AAT.IDX_NGAS_SUBSCRIBERS_SUBSCR_ID) violated and HTTP code 400

idiom with open(...) as fo: will open a file in text mode so read() returns string but bytes are needed.
In addition print(traceback.print_exc()) makes no sense since traceback.print_exc() already prints the stack trace and returns nothing

--- a/src/ngamsUtils/ngamsUtils/ngasXSyncTool.py
+++ b/src/ngamsUtils/ngamsUtils/ngasXSyncTool.py
@@ -528,7 +528,7 @@ def initialize(param_dict):
     else:
         try:
             hash_md5 = hashlib.md5()
-            with open(param_dict[PAR_FILE_LIST]) as fo:
+            with open(param_dict[PAR_FILE_LIST], mode='rb') as fo:
                 hash_md5.update(fo.read())
             session_id = hash_md5.hexdigest()
         except Exception as e:
@@ -1337,7 +1337,7 @@ def main():
             if str(e) == "0":
                 sys.exit(0)
             print("\nProblem executing the tool:\n\n{:s}\n".format(str(e)))
-            print(traceback.print_exc())
+            traceback.print_exc()
             sys.exit(1)

PEP 594 plans on removing smtpd from the standard library, it's probably worth working out what's needed to replace its usage within ngas.

The ngamsServer and ngamsDaemon both create PID files but with different paths. The ngamsServer has a command line option -force which the help description says Force the start of the server, even if a PID file is present. The allows us to recover from a crash. The PID file is stored at /srv/ngas1/.aat-ngas-2:8001.pid.
However, the ngamsDaemon does not provide any support for cleaning up bad PID files which means the user must manually delete the PID typically found at /srv/ngas1/var/run/ngamsDaemon.pid. This is quite inconvenient for obvious reasons.
Can the ngamsDaemon also support the -force command line option? This would make recovering from a crash much easier and efficient.

(found running mypy):
In ngamsCore/ngamsLib/logutils.py local variable "text" starts out as a list of text fragments but is then reused as a string of those concatenated text fragments. Better:

diff --git a/src/ngamsCore/ngamsLib/logutils.py b/src/ngamsCore/ngamsLib/logutils.py
index a471da83..afbe8445 100644
--- a/src/ngamsCore/ngamsLib/logutils.py
+++ b/src/ngamsCore/ngamsLib/logutils.py
@@ -78,13 +78,13 @@ class LogDefHolder(object):

             # Get the Log Text
             tmpNodeList = node.getElementsByTagName("LogText")
-            text = []
+            list_text_fragments = []
             for nd in tmpNodeList[0].childNodes:
                 if (nd.nodeType == node.TEXT_NODE):
-                    text.append(nd.data.strip(" \n"))
+                    list_text_fragments.append(nd.data.strip(" \n"))
             # Remove newline characters and ensure that there is no
             # sequence of blanks.
-            text = ' '.join(text)
+            text = ' '.join(list_text_fragments)
             text = text.replace("\n", "")
             text = re.sub(r"\s *", " ", text)
             ltext = text

The python NGAS archive client application cannot use the --servers command line option value. At line 90 in ngamsPClient.py an exception is raised when the servers and host/port command line options are set. As the host and port command line values are assigned default values this means an exception will always be raised when the --servers is used.

The simple solution would be to remove the default values for the host/port command line options. Otherwise grant the servers command line option precedence when it is set.

I am reproducing the comment I added on issue-51...
@rtobar I got a message from Eisuke. He reported that data-check thread has scanned all the files under */volume*/ and files found under the bad-files directory were reported as NOT REGISTERED FILES
He suggests that "it may be useful for admin to know there is unprocessed files under bad-files but perhaps using another category, e.g. BAD-FILES FOUND ON STORAGE DISKS is better than NOT REGISTERED FILES
What do you think?

As noted by @smclay, and years ago by myself, the REGISTER command doesn't always return a failure to the user when file registration files. In particular, some errors in the _registerExec function are simply logged and then reported via email, but not reported back to the client originating the request. _registerExec can be executed either during the original HTTP request, or in the background, and therefore it was probably originally thought that raising exceptions not always resulted in user-visible errors.

This situation should be improved, with users receiving errors when they happen.

NGAS is doing the right thing, but the message returned could be better, describing that the issue is really that there was no matching mimetype, rather than the reported error about getHostIdList() not being an attribute of NoneType. Workaround is: get my mimetypes right :)

I can reproduce In v11.0.2, with the following qarchive command:
curl -X POST -i -H "Content-Type: application/mimetype_not_existing" --data-binary @test.fits "http://ngashost:7777/QARCHIVE?filename=test.fits"

Returns:
...
Message="'NoneType' object has no attribute 'getHostIdList'" State="ONLINE" Status="FAILURE" SubState="IDLE" Version="11.0.2/2018-11-07T11:00:00"/>

In the log file:
2019-10-15T08:37:36.007 [29022] [ R-6] [ INFO] ngamsServer.ngamsCmdHandling#_get_module:74 Received command: QARCHIVE
2019-10-15T08:37:36.007 [29022] [ R-6] [ERROR] ngamsServer.ngamsServer#reqCallBack:1678 Error while serving request
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/ngamsServer/ngamsServer.py", line 1662, in reqCallBack
method, path, headers)
File "build/bdist.linux-x86_64/egg/ngamsServer/ngamsServer.py", line 1756, in handleHttpRequest
ngamsCmdHandling.handle_cmd(self, reqPropsObj, httpRef)
File "build/bdist.linux-x86_64/egg/ngamsServer/ngamsCmdHandling.py", line 63, in handle_cmd
msg = _get_module(srvObj, reqPropsObj).handleCmd(srvObj, reqPropsObj, httpRef)
File "build/bdist.linux-x86_64/egg/ngamsServer/commands/qarchive.py", line 70, in handleCmd
do_probe=False, try_to_proxy=True)
File "build/bdist.linux-x86_64/egg/ngamsServer/ngamsArchiveUtils.py", line 1004, in archiveInitHandling
srvObj.getCfg().getStreamFromMimeType(mimeType).getHostIdList()):
AttributeError: 'NoneType' object has no attribute 'getHostIdList'
2019-10-15T08:37:36.007 [29022] [ R-6] [ INFO] ngamsServer.ngamsServer#send_status:335 Returning status FAILURE with message 'NoneType' object has no attribute 'getHostIdList' and HTTP code 400
2019-10-15T08:37:36.008 [29022] [ R-6] [ INFO] ngamsServer.ngamsServer#send_data:304 Sending 366 bytes of data of type text/xml and headers {}

Looking at the code, the part that fails is in ngamsArchiveUtils.py:
if (try_to_proxy and
srvObj.getCfg().getStreamFromMimeType(mimeType).getHostIdList()):
...

getStreamFromMimeType(mimeType) returns None, presumably because the mimeType was not found in the NGAS config, and then getHostIdList() is called on that returned None object. So maybe check for the return value of getStreamFromMimeType(mimeType) first and only if it is not None to call getHostIdList().

At the moment the ngas_hosts.cluster_name database value is set by default to the ngas_hosts.host_id value. As far as I can see if the administrator would like to change the cluster name they need to set it directly in the database using SQL. I think it would be better if this value could be set from the XML configuration file which should be a trivial feature to implement.

Currently the subscription service (as a client) can only use HTTP, not HTTPS. I'm planning on using requests to handle the whole HTTPS ecosystem (so that I don't need to worry about certificates from a client's perspective), but currently there's no support in requests (nor in its underlying stack) to use sendfile (also, sendfile over https requires kernel support, and wrappers around the kernel, so that's a whole different thing). I'm going to implement this via only using requests for HTTPS, and putting all HTTPS through requests (i.e. no sendfile over HTTPS), so there shouldn't be any breaking changes. Would a PR containing this be fine on top of master, or would it need to go into next?

I have been testing the ngamsArchiveClient. It seems to be running well. However, when I check the log file the log message formatting is wrong. It appears to write {timestamp} [{log-level}] [{process-id}] {timestamp} [{log-level}] [{process-id}] {timestamp} [{log-level}] [{process-id}] .... The log message and new line character appear to be truncated. Here is a sample...

2021-01-22T10:01:45.614 [INFO] [140403126805168] 2021-01-22T10:01:45.615 [INFO] [140403126805168] 2021-01-22T10:01:45.615 [INFO] [140403126805168] 2021-01-22T10:01:45.615 [INFO] [140403126805168] 2021-01-22T10:01:45.615 [INFO] [140403126805168] 2021-01-22T10:01:45.616 [INFO] [140403126805168] 2021-01-22T10:01:45.616 [INFO] [140403126805168] 2021-01-22T10:01:45.616 [INFO] [140403126805168]

The REGISTER command works via plug-ins that take care of different mime-types. Although some configuration files refer to an ngamsGenRegPlugIn it seems no such file has ever been present in the codebase (as initially noted by @smclay, then confirmed by myself). We should provide a default., generic plug-in for the REGISTER command so users can configure their systems to use it instead of having to write their own.

During the work done in #19 it was found by @gsleap that no matter the combination of parameters given to bbcp, it apparently always resorted to using hostnames (and fully qualified domain names) as the sole basis for establishing the connection between the source and the target nodes. This is not enough for certain scenarios, where two machines have multiple, independent networks paths that can be taken depending on the interface being addressed.

Consider the following scenario, which is similar to the deployment used in the tests described in #19:

          Host A                                        Host B
      +-------------+            1 Gb              +-------------+
      |        eth0 |<---------------------------->| eth0        |
      | IP: 1.1.1.1 |                              | IP: 1.1.1.2 |
      |             |                              |             |
      |             |            10 Gb             |             |
      |        eth1 |<---------------------------->| eth1        |
      | IP: 2.2.2.2 |                              | IP: 2.2.2.3 |
      |             |                              |             |
      |             |                              |             |
      | NGAS client |                              | NGAS server |
      |    bbcp SRC |                              | bbcp SINK   |
      +-------------+                              +-------------+

The NGAS server running in B is listening on eth1, the 10 Gb interface. When the BBCPARC command comes in from A we generate and execute this command in B:

bbcp .... 2.2.2.2:/path/to/source/file 2.2.2.3:/path/to/ngas/staging/file

By using the specific IPs in the source and target specifications we expect bbcp to explicitly use the 10 Gb interface for the data transfer.

The command starts the SRC and SINK copies of bbcp in A and B respectively. bbcp however seems to use exclusively the hosts' names as the main bit of information to establish the connection between SRC and SINK, and because A and B resolve to the 1.1.1.X addresses, the 1Gb link is used for the bbcp data transfers. This behavior seems to be same regardless of the direction of the connection establishment (i.e. the -z option) and whether name resolution (i.e., the -n option) is used, but this should be tested thoroughly.

This problem was initially investigated in #19, but then it was decoupled into a new issue to separate it from the original problem reportedin #19, which has been fixed.

An NGAS system can be configured to accept a number of MIME types, from which some might not be configured to be handled with a REGISTER plug-in. When a client tries to register a file with one of these half-configured MIME types the server responds with a simple "<mime-type>" message, where <mime-type> is the MIME type in question. This is of course not very informative, and a better error message explaining the full situation (e.g., "Mime type has no associated plug-in") would improve things.

It was suggested to me by one of the ARC system administrators that it would be better to move the bad-files sub-directory under the volumes directory. Similar to the staging area directory.

At the moment NGAS creates a bad-files directory /srv/ngas1/bad-files and the staging area directory /srv/ngas1/volume1/staging. It would be better for consistency if NGAS were to create the bad-files directory in the same volume directory as the staging area directory. For example, /srv/ngas1/volume1/bad-files.

I believe the main advantages are...

1. More consistent
2. It is easier to trace which files failed to archive from which volumes
3. It is more flexible for creating NGAS volume mount points

I believe this last point is of particular interest to our ARC colleague because they are mounting NGAS volumes on a SAN.

I created a test where I archive files specifying different checksum variants...

ngas-cmd -a localhost -p 8001 archive filename=test_sample_crc32.txt crc_variant=crc32
ngas-cmd -a localhost -p 8001 archive filename=test_sample_crc32c.txt crc_variant=crc32c
ngas-cmd -a localhost -p 8001 archive filename=test_sample_crc32z.txt crc_variant=crc32z

Only the file archived using crc32c is successfully pushed to the a subscriber. Looking at the logs from the subscriber I see the following error messages...

2021-03-06T17:54:15.181 [20092] [     R-510] [  INFO] ngamsServer.ngamsArchiveUtils#_dataHandler:1083 NGAMS_INFO_ARCHIVING_FILE:4026:INFO: Archiving file with URI: test_sample_crc32z.txt
2021-03-06T17:54:15.199 [20092] [     R-510] [  INFO] ngamsServer.ngamsArchiveUtils#_dataHandler:1101 Archiving file: test_sample_crc32z.txt with mime-type: text/plain
2021-03-06T17:54:15.200 [20092] [SUBSCRIBER] [WARNIN] ngamsServer.ngamsSrvUtils#_create_remote_subscriptions:140 Unsuccessful NGAS XML response. Status: FAILURE, message: ORA-00001: unique constraint (NGAS05_AAT.IDX_NGAS_SUBSCRIBERS_SUBSCR_ID) violated. Will try again later
2021-03-06T17:54:15.202 [20092] [     R-510] [WARNIN] ngamsServer.ngamsArchiveUtils#cleanUpStagingArea:1117 Removing Staging File: /srv/ngas1/volume1/staging/NGAMS_TMP_FILE___DAINVetest_sample_crc32z.txt
2021-03-06T17:54:15.203 [20092] [     R-510] [WARNIN] ngamsServer.ngamsArchiveUtils#cleanUpStagingArea:1117 Removing Staging File: /srv/ngas1/volume1/staging/nNpIcT-test_sample_crc32z.txt
2021-03-06T17:54:15.203 [20092] [     R-510] [WARNIN] ngamsServer.ngamsArchiveUtils#cleanUpStagingArea:1117 Removing Staging File: /srv/ngas1/volume1/staging/NGAMS_TMP_FILE___DAINVetest_sample_crc32z.txt.pickle
2021-03-06T17:54:15.203 [20092] [     R-510] [WARNIN] ngamsServer.ngamsArchiveUtils#cleanUpStagingArea:1117 Removing Staging File: /srv/ngas1/volume1/staging/nNpIcT-test_sample_crc32z.txt.pickle
2021-03-06T17:54:15.205 [20092] [     R-510] [ ERROR] ngamsServer.ngamsServer#reqCallBack:1743 Error while serving request
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsServer.py", line 1727, in reqCallBack
    method, path, headers)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsServer.py", line 1822, in handleHttpRequest
    ngamsCmdHandling.handle_cmd(self, reqPropsObj, httpRef)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsCmdHandling.py", line 63, in handle_cmd
    msg = _get_module(srvObj, reqPropsObj).handleCmd(srvObj, reqPropsObj, httpRef)
  File "/usr/lib/python2.7/site-packages/ngamsServer/commands/archive.py", line 69, in handleCmd
    do_replication=True)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsArchiveUtils.py", line 1039, in dataHandler
    do_replication=do_replication, transfer=transfer)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsArchiveUtils.py", line 1138, in _dataHandler
    rfile, skip_crc=skip_crc, transfer=transfer)
  File "/usr/lib/python2.7/site-packages/ngamsServer/ngamsArchiveUtils.py", line 249, in archive_contents_from_request
    raise Exception(msg)
Exception: Checksum error for file test_sample_crc32z.txt, local crc = 3225417148, but remote crc = 972511042
2021-03-06T17:54:15.205 [20092] [     R-510] [  INFO] ngamsServer.ngamsServer#send_status:350 Returning status FAILURE with message Checksum error for file test_sample_crc32z.txt, local crc = 3225417148, but remote crc = 972511042 and HTTP code 400

I also receive the following notification email message...

Notification Message:


ACCUMULATED NOTIFICATION MESSAGES:

NOTE: Distribution of retained Email Notification Messages forced at Offline

--MESSAGE#000001/2021-03-06T17:54:15.204----------
NGAMS_ER_ARCHIVE_PUSH_REQ:4011:ERROR: Problem occurred handling Archive Push Request! URI: test_sample_crc32z.txt. Error: Checksum error for file test_sample_crc32z.txt, local crc = 3225417148, but remote crc = 972511042.
--END-----------------------------------------


Note: This is an automatically generated message

On startup NGAS creates a janitor process. The janitor process reports and error when it tries to archive a rotated log file. The error occurs because the NGAS HTTP server is not ready to handle requests. Here are the log messages...

2020-08-20T09:27:49.310 [ 1881] [JANITOR-PR] [  INFO] ngamsServer.janitor.expired_data_cleaner#run:54 Checking/cleaning up processing directory: /srv/ngas1/./processing
2020-08-20T09:27:49.310 [ 1881] [JANITOR-PR] [  INFO] ngamsServer.janitor.expired_data_cleaner#run:54 Checking/cleaning up subscription backlog buffer: /srv/ngas1/./subscr-back-log
2020-08-20T09:27:49.311 [ 1881] [JANITOR-PR] [  INFO] ngamsServer.janitor.expired_data_cleaner#run:54 Checking/cleaning up NGAS tmp directory: /srv/ngas1/tmp
2020-08-20T09:27:49.314 [ 1881] [JANITOR-PR] [  INFO] ngamsPClient.ngamsPClient#archive:142 Archiving file with URI: file:///srv/ngas1/log/LOG-ROTATE-2020-08-20T09:23:53.255.nglog
2020-08-20T09:27:49.315 [ 1881] [JANITOR-PR] [  INFO] ngamsLib.ngamsHttpUtils#_http_response:125 About to GET to 127.0.0.1:8001//ARCHIVE?async=0&no_versioning=0&filename=file%3A%2F%2F%2Fsrv%2Fngas1%2Flog%2FLOG-ROTATE-2020-08-20T09%3A23%3A53.255.nglog&mime_type=ngas%2Fnglog
2020-08-20T09:27:49.317 [ 1881] [JANITOR-PR] [ ERROR] ngamsServer.janitor.main#JanitorCycle:66 Unexpected error in janitor plug-in run
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ngamsServer/janitor/main.py", line 62, in JanitorCycle
    p(srvObj, stopEvt)
  File "/usr/lib/python2.7/site-packages/ngamsServer/janitor/rotated_logfiles_handler.py", line 76, in run
    file_uri, 'ngas/nglog')
  File "/usr/lib/python2.7/site-packages/ngamsPClient/ngamsPClient.py", line 153, in archive
    return self.get_status(cmd, pars=pars)
  File "/usr/lib/python2.7/site-packages/ngamsPClient/ngamsPClient.py", line 693, in get_status
    resp, host, port = self._get(cmd, pars, hdrs)
  File "/usr/lib/python2.7/site-packages/ngamsPClient/ngamsPClient.py", line 719, in _get
    resp = self._do_get(host, port, cmd, pars, hdrs)
  File "/usr/lib/python2.7/site-packages/ngamsPClient/ngamsPClient.py", line 756, in _do_get
    timeout=self.timeout, auth=self.basic_auth
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsHttpUtils.py", line 303, in httpGetUrl
    pars=allpars, hdrs=hdrs, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsHttpUtils.py", line 273, in httpGet
    pars=pars, hdrs=hdrs, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsHttpUtils.py", line 127, in _http_response
    _connect(conn)
  File "/usr/lib/python2.7/site-packages/ngamsLib/ngamsHttpUtils.py", line 63, in _connect
    conn.connect()
  File "/usr/lib64/python2.7/httplib.py", line 833, in connect
    self.timeout, self.source_address)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused
2020-08-20T09:27:49.324 [ 1881] [JANITOR-PR] [  INFO] ngamsServer.janitor.main#janitorThread:131 Janitor Thread executed - suspending for 86400 [s] ...
2020-08-20T09:27:49.723 [ 1869] [DATA-CHECK] [  INFO] ngamsServer.ngamsDataCheckThread#dataCheckThread:787 Data Check Thread starting iteration ...
2020-08-20T09:27:49.741 [ 1869] [DATA-CHECK] [  INFO] ngamsServer.ngamsDataCheckThread#get_disks_to_check:678 Will check 0 disks that are mounted in this system
2020-08-20T09:27:49.743 [ 1869] [DATA-CHECK] [  INFO] ngamsServer.ngamsDataCheckThread#collect_files_on_disk:205 Found 0 files to check in 0 disks
2020-08-20T09:27:49.858 [ 1869] [MainThread] [  INFO] ngamsServer.ngamsServer#serve:2177 NG/AMS HTTP Server ready

EDIT by rtobar to format error message

When trying to run a server with a subscription on python 3

I am running a mirroring test from an legacy NGAS cluster to a NGAS github cluster. I get the following logging error message...

2021-12-12T11:33:42.993 [26176] [Thread-314] [ ERROR] ngamsPlugIns.ngamsCmd_MIRREXEC#process_mirroring_tasks:500 Failed to fetch test_sample_1000_0775.txt
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ngamsPlugIns/ngamsCmd_MIRREXEC.py", line 496, in process_mirroring_tasks
    ngamsCmd_MIRRARCHIVE.handleCmd(ngams_server, request_properties)
  File "/usr/lib/python3.6/site-packages/ngamsPlugIns/ngamsCmd_MIRRARCHIVE.py", line 117, in handleCmd
    __handle_command(ngams_server, request_properties)
  File "/usr/lib/python3.6/site-packages/ngamsPlugIns/ngamsCmd_MIRRARCHIVE.py", line 156, in __handle_command
    staging_filename, start_byte)
  File "/usr/lib/python3.6/site-packages/ngamsPlugIns/ngamsCmd_MIRRARCHIVE.py", line 105, in save_in_staging_file
    block_size, start_byte)
  File "/usr/lib/python3.6/site-packages/ngamsPlugIns/ngamsCmd_HTTPFETCH.py", line 185, in save_to_file
    raise ngamsFailedDownloadException.FailedDownloadException(msg)
ngamsPlugIns.ngamsFailedDownloadException.FailedDownloadException: 'checksum mismatch: source=-942573743, received=3352393553'

This only happens when I run the NGAS github cluster on python 3. When running on python 2 the checksum calculations match.

Hi, I am wondering if it is possible to query NGAS for FITS files with matching key/value pairs in their headers? Or should this info be encoded into the filename?

Thanks!

We have some test NGAS servers. They have a limited disk capacity. We use them intensively for testing other software applications. In order to prevent the disks from being completely filled up I have a utility script that runs once per weeks and discards all files more than one day old. However, the tests have been more intensive than normal and the disks were filled to capacity. This highlighted an issue.

When the disks reach capacity they are marked as completed. My utility script then discarded old files freeing up lots of disk space. However, NGAS does not automatically update the completed status. I have to manually edit the database table. I think it would be very useful if NGAS was smart enough to check and update the completion status when files are discarded. Perhaps the janitor or data check thread could monitor the completion status.

When issuing a BBCPARC command:
http://ngashost:7777/BBCPARC?filename=user%40hostwithfile%3A%2Fdata%2F1247842824_20190722150006_ch114_000.fits&bnum_streams=12&mime_type=application%2Fx-mwa-fits

NGAS server returns 400 BAD REQUEST, which error as shown in log file as below:
2019-12-06T05:01:03.292 [ 5303] [ R-65] [ INFO] ngamsServer.ngamsServer#handleHttpRequest:1696 Handling HTTP request: client_address=('192.168.120.204', 43008) - method=GET - path=|BBCPARC?filename=user%40hostwithfile%3A%2Fdata%2F1247842824_20190722150006_ch114_000.fits&bnum_streams=12&mime_type=application%2Fx-mwa-fits|
2019-12-06T05:01:03.293 [ 5303] [ R-65] [ INFO] ngamsServer.ngamsCmdHandling#_get_module:74 Received command: BBCPARC
2019-12-06T05:01:03.293 [ 5303] [ R-65] [ INFO] ngamsServer.ngamsArchiveUtils#_dataHandler:1055 Handling archive pull request
2019-12-06T05:01:03.294 [ 5303] [ R-65] [ ERROR] ngamsServer.ngamsServer#reqCallBack:1633 Error while serving request
Traceback (most recent call last):
File "/usr/lib/python3.6/urllib/request.py", line 1474, in open_local_file
stats = os.stat(localfile)
FileNotFoundError: [Errno 2] No such file or directory: '/user@hostwithfile:/data/1247842824_20190722150006_ch114_000.fits'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsServer.py", line 1617, in reqCallBack
method, path, headers)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsServer.py", line 1711, in handleHttpRequest
ngamsCmdHandling.handle_cmd(self, reqPropsObj, httpRef)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsCmdHandling.py", line 63, in handle_cmd
msg = _get_module(srvObj, reqPropsObj).handleCmd(srvObj, reqPropsObj, httpRef)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/commands/bbcparc.py", line 182, in handleCmd
transfer=bbcp_transfer)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsArchiveUtils.py", line 1033, in dataHandler
do_replication=do_replication, transfer=transfer)
File "/home/mwa/ngas_rt/lib/python3.6/site-packages/ngamsServer-11.0-py3.6.egg/ngamsServer/ngamsArchiveUtils.py", line 1062, in _dataHandler
handle = urlrequest.urlopen(url)
File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/lib/python3.6/urllib/request.py", line 1452, in file_open
return self.open_local_file(req)
File "/usr/lib/python3.6/urllib/request.py", line 1491, in open_local_file
raise URLError(exp)
urllib.error.URLError: <urlopen error [Errno 2] No such file or directory: '/user@hostwithfile:/data/1247842824_20190722150006_ch114_000.fits'>

NOTE: this is not a showstopper, as I can easily use QARCHIVE, but I thought I would try out bbcparc first.

Remove deprecated threading.Thread function calls from src/ngamsCore/ngamsLib/ngamsThreadGroup.py
(found by runnning mypy on src/ngamsCore).

references:
https://docs.python.org/3/library/threading.html#thread-objects
spotify/luigi#2939

diff --git a/src/ngamsCore/ngamsLib/ngamsThreadGroup.py b/src/ngamsCore/ngamsLib/ngamsThreadGroup.py
index c039ef73..647bfcbc 100755

--- a/src/ngamsCore/ngamsLib/ngamsThreadGroup.py
+++ b/src/ngamsCore/ngamsLib/ngamsThreadGroup.py
@@ -153,7 +153,7 @@ class ngamsThreadGroup:
             thrId = "%s-%d" % (id, n)
             thrObj = threading.Thread(None, self.__threadEncapsulator,
                                       thrId, args)
-            thrObj.setDaemon(0)
+            thrObj.daemon = False
             thrObj.start()
             self.__threadHandles.append(thrObj)

@@ -194,7 +194,7 @@ class ngamsThreadGroup:
         """
         activeThreads = 0
         for threadHandle in self.__threadHandles:
-            if (threadHandle.isAlive()): activeThreads += 1
+            if (threadHandle.is_alive()): activeThreads += 1
         return activeThreads


@@ -349,7 +349,7 @@ class ngamsThreadGroup:
                           "to terminate"
                     raise Exception(msg)
             thrWaitingList[0].join(curTimeout)
-            if (not thrWaitingList[0].isAlive()): del thrWaitingList[0]
+            if (not thrWaitingList[0].is_alive()): del thrWaitingList[0]
             if (thrWaitingList == []):
                 return self

It would appear that NGAS only supports a single file extension per mime-type. It would be desirable if NGAS could support multiple files extensions with the same mime-type. For example bin, dat and exe could all be application/octet-stream, htm and html could be text/html.

The NgamsStatus XML response returned by NGAS contains an invalid link to the corresponding DTD file, which makes the XML validation fail if we use a validating XML parser.

Example:

curl http://ta20:7777/STATUS?file_id=ADP.2023-05-25T13:34:00.575

<?xml version="1.0" ?>
<!DOCTYPE NgamsStatus SYSTEM "http://ta20.hq.eso.org:7777/RETRIEVE?internal=ngamsStatus.dtd">
...

curl "http://ta20.hq.eso.org:7777/RETRIEVE?internal=ngamsStatus.dtd"

<?xml version="1.0" ?>
<!DOCTYPE NgamsStatus SYSTEM "http://ta20.hq.eso.org:7777/RETRIEVE?internal=ngamsStatus.dtd">
<NgamsStatus>
  <Status Date="2023-06-29T14:49:38.376" HostId="ta20:7777" Message="ng_log, cfg and internal parameters not supported anymore" State="ONLINE" Status="FAILURE" SubState="IDLE" Version="12.0/2022-11-29T07:00:00"/>

I can think of two possible approaches for fixing this issue:

1. remove the DOCTYPE declaration from the response
2. re-enable DTD retrieval

I found this list of new features for NGAS: https://confluence.icrar.uwa.edu.au/display/NUG/NGAS+New+Feature+Discussions+and+Requirements. Is this the most recent discussion? If no, is there a roadmap that includes moving to python 3?

This function is supposed to return a tuple of five string values. However since the resource file is written by an end user there is no guarantee that any of the parameters which get_db_parameters() tries to find is defined by the user. We just had the case that property DbInterface was not set in such rc file and it lead to a really hard to understand runtime exception when executing utility ngas-xsync-tool. I would argue that get_db_parameters() shall fail immediately with a clear error message whenever one of the properties that get_db_parameters() tries to read from an resource file is missing.

The code as a mix of print functions and print statements, old style Exception handling and more.

I get the following bug when attempting to use the --server command line option for the NGAMS python client

$ /usr/bin/ngams3PClient --servers "my-server:7777" --status /path/to/response.xml --timeout 5 --file-uri /path/to/file.txt --mime-type text/data ARCHIVE
Traceback (most recent call last):
  File "/usr/bin/ngams3PClient", line 11, in <module>
    load_entry_point('ngamsPClient==11.0', 'console_scripts', 'ngamsPClient')()
  File "/usr/lib/python3.6/site-packages/ngamsPClient/ngamsPClient.py", line 887, in main
    servers = [(host, int(port)) for s in opts.servers.split(',') for host,port in s.split(':')]
  File "/usr/lib/python3.6/site-packages/ngamsPClient/ngamsPClient.py", line 887, in <listcomp>
    servers = [(host, int(port)) for s in opts.servers.split(',') for host,port in s.split(':')]
ValueError: too many values to unpack (expected 2)

attribute __ioTime is initialized with 0 (Integer) but later only ever assigned float values.
(found running mypy on src/ngamsCore)

diff --git a/src/ngamsCore/ngamsLib/ngamsDapiStatus.py b/src/ngamsCore/ngamsLib/ngamsDapiStatus.py
index beb136dd..be2f08db 100644
--- a/src/ngamsCore/ngamsLib/ngamsDapiStatus.py
+++ b/src/ngamsCore/ngamsLib/ngamsDapiStatus.py
@@ -53,7 +53,7 @@ class ngamsDapiStatus:
         self.__compression       = ""
         self.__relPath           = ""
         self.__slotId            = ""
-        self.__ioTime            = 0
+        self.__ioTime            = 0.0
         self.__fileExists        = -1
         self.__completeFilename  = ""
         self.crc = None

The query command does not support the old ignore column in ngas_files
Example: curl http://<host>:<port>/QUERY?query=files_list
Error: Invalid column name 'file_ignore'

(found running mypy)
in src/ngamsCore/ngamsLib/ngamsCore.py function isoTime2Secs() returns a float value according to its docstring but only achieves this by relying on implicit behavior for rounding (many intermadeiate values are Integers) and implicit type conversion

diff --git a/src/ngamsCore/ngamsLib/ngamsCore.py b/src/ngamsCore/ngamsLib/ngamsCore.py
index 5889bf1c..9cb4f4ee 100644
--- a/src/ngamsCore/ngamsLib/ngamsCore.py
+++ b/src/ngamsCore/ngamsLib/ngamsCore.py
@@ -640,11 +640,11 @@ def isoTime2Secs(isoTime):
     timeVector = timePart.split(":")
     hours = int(timeVector[0])
     mins = int(timeVector[1])
-    secs = 0
+    secs = 0.0
     if len(timeVector) > 2:
         secs = float(timeVector[2])

-    return days*24*3600 + hours*3600 + mins*60 + secs
+    return float(days*24*3600 + hours*3600 + mins*60) + secs

NGAS supports three checksum variants: crc32, crc32c and crc32z
As described in the online documentation and in the code, the first variant is "inconsistent" because it returns a signed/unsigned result depending on the platform.
Rather than supporting an inconsistent variant it would be more useful to support a backward-compatible one which consistently returns a signed CRC as in Python 2. This would be of great help when upgrading an existing archive from Python 2 to Python 3.
My preference would be to change crc32 to be backward compatible instead of inconsistent but I'm also OK with adding a new variant.
BTW ngamsCrc32.c returns a signed integer as well.

The code is doing some unnecessary string gymnastics, so it can be slightly simplified.

In Oracle "comment" is a reserved word, "archive" is a keyword and "type" is a "PL/SQL" reserved word (see https://docs.oracle.com/database/121/ZZPRE/appb.htm#ZZPRE959). Could we change the names of the following table columns ngas_subscr_queue.comment, ngas_disks.archive and ngas_disks.type in a future release?