Unable to push basic FITS file with NGAS QUARCHIVE: "Keyword 'ARCFILE' not found." about ngas

[rtobar]

Hi @danjampro,

This is "expected" behavior in the sense that we know this is what the default FITS plugin does, but is of course unexpected for new users. I think this is mostly historical: when NGAS was originally developed at ESO, they wanted to have a FITS keyword to specify a filename, and use that as the final filename on storage. Note that because of this, in theory any plug-in can compute filenames in different ways, instead of relying in the filename HTTP parameter, which serves as the default.

There are a few ways to go around this: you can take a copy of the default FITS plug-in and modify it to not take into account the ARCFILE keyword, and use that plugin instead; you could also try to archive the file with mime_type=application/octet-stream which would work, but you'd loose the mime type information; or you could try to add a new parameter to the FITS plug-in to skip checking for the ARCFILE keyword and use the incoming filename instead as the name of the file on disk.

[danjampro]

Hi @rtobar, thanks for your response.

I just successfully archived a file with octet-stream. Is there a way to preserve the mime type with this approach? It would be good to avoid creating our own fork of NGAS.

Edit: Is preserving the mime type important? How does NGAS use it?

[rtobar]

No, the mime_type=application/octet-stream simply forces NGAS to use a different plug-in, as plug-ins are fully mime-type-dependent. So yes, mime-types are fairly central to the operation of the server; not only are they used for picking a archiving plug-in, but storage can also be organized around mime types, plus other details.

To adjust for your use case (a very valid one!) I just added a new ignore_arcfile option for the FITS plug-in to ignore the ARCFILE keyword (off by default to preserve current behavior), could you give it a try? It's in the new fits_optional_arcfile branch. Adding an ignore_arcfile=1 HTTP parameter should be enough to activate the functionality.

[danjampro]

Hit @rtobar, thanks for adding this modification. I tested it with ignore_arcfile=1 but I get the error:

dc_ngas_cache_ | Traceback (most recent call last):
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsServer-11.0-py3.7.egg/ngamsServer/ngamsServer.py", line 1634, in reqCallBack
dc_ngas_cache_ | method, path, headers)
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsServer-11.0-py3.7.egg/ngamsServer/ngamsServer.py", line 1728, in handleHttpRequest
dc_ngas_cache_ | ngamsCmdHandling.handle_cmd(self, reqPropsObj, httpRef)
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsServer-11.0-py3.7.egg/ngamsServer/ngamsCmdHandling.py", line 63, in handle_cmd
dc_ngas_cache_ | msg = get_module(srvObj, reqPropsObj).handleCmd(srvObj, reqPropsObj, httpRef)
dc_ngas_cache | File "/usr/local/lib/python3.7/site-packages/ngamsServer-11.0-py3.7.egg/ngamsServer/commands/qarchive.py", line 80, in handleCmd
dc_ngas_cache_ | do_replication=False)
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsServer-11.0-py3.7.egg/ngamsServer/ngamsArchiveUtils.py", line 1033, in dataHandler
dc_ngas_cache_ | do_replication=do_replication, transfer=transfer)
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsServer-11.0-py3.7.egg/ngamsServer/ngamsArchiveUtils.py", line 1168, in dataHandler
dc_ngas_cache | plugin_result = plugInMethod(srvObj, reqPropsObj)
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsPlugIns-11.0-py3.7.egg/ngamsPlugIns/ngamsFitsPlugIn.py", line 306, in ngamsFitsPlugIn
dc_ngas_cache_ | uncomprSize, archFileSize, mime, compression, crc = compress(reqPropsObj, parDic)
dc_ngas_cache_ | File "/usr/local/lib/python3.7/site-packages/ngamsPlugIns-11.0-py3.7.egg/ngamsPlugIns/ngamsFitsPlugIn.py", line 265, in compress
dc_ngas_cache_ | return uncomprSize, archFileSize, mime, compression, crc
dc_ngas_cache_ | UnboundLocalError: local variable 'crc' referenced before assignment

[rtobar]

Hi @danjampro, thanks for testing these changes. In fact you just hit a different error, this time related to the configuration of compression on the server-side. I again added a test to reproduce it, and also a patch to fix it. Would you mind pulling the latest changes from the fits_optional_arcfile branch and try again?

[danjampro]

@rtobar I seem to have a talent for that!

Just tried it and looks like it worked. Here is the response:

Looks good to me - thanks a lot for your help, much appreciated.

Edit: Will this be merged with the master branch?

[rtobar]

@danjampro yes, I just did the merge so I'm closing this issue now.