XML response contains invalid link to NgamsStatus DTD about ngas

[rtobar]

@szampier thanks for reporting this, and sorry for not being able to come back earlier.

As noted in the commit message above, we disabled the internal, ng_log and cfg parameters a while ago because of the security risk they represented. I've special-cased the internal=ngamsStatus.dtd value now though, and added a test that checks this is working as expected. Please let me know if this works for you now and I'll merge back to master.

This might or might not open another can of worms in terms of invalid XML being returned... hopefully nothing too bad :)

[szampier]

Hi @rtobar , thanks for looking into this. I've verified that it's now possible to retrieve ngamsStatus.dtd but this DTD refers to other internal DTDs located at DUMMYHOST:DUMMYPORT...
So the error is now: "MalformedURLException" (Java) or "Invalid URI" (python lxml).

$ curl http://127.0.0.1:7777/RETRIEVE?internal=ngamsStatus.dtd
<?xml version="1.0" encoding="UTF-8"?>
<!ENTITY % XmlStd SYSTEM "http://DUMMYHOST:DUMMYPORT/RETRIEVE?internal=XmlStd.dtd">
%XmlStd;
<!ENTITY % NgamsInternal SYSTEM 
           "http://DUMMYHOST:DUMMYPORT/RETRIEVE?internal=ngamsInternal.dtd">
%NgamsInternal;
...

[rtobar]

Mmmm... moreover, the contents of the DTD document are non-standard (e.g., see the third line, which reads %XmlStd;, definitely not XML!). There was some code in NGAS that did some custom substitutions, but from what I can dig this code was never invoked, not even in the first version of the code we imported in our git repo, so I have no idea how this would have ever worked.

@szampier could you show me how this looks for for the ESO version? If anything, I'd try to match that behaviour. @smclay might want to chip in as well from the ALMA side, in case they're using this feature.

Sorry, I stand corrected, this was me not knowing enough about DTD (never had to deal with it before, I think it had been superseeded by XSD by the time I starting dealing with XML). The line I pointed out is a https://www.w3.org/TR/xml11/#NT-PEReference. It all looks good then, and it should all work if I change those URLs to point to the server.

[szampier]

Hi @rtobar, below is a diff between VLT-NGAS (right) and ICRAR (left) ngamsStatus.dtd. Note that the VLT one points to a public document on the ESO pages, which is also not optimal IMO.

diff ngamsStatus.dtd ngamsStatus-icrar.dtd 
2c2
< <!ENTITY % XmlStd SYSTEM "http://www.eso.org/projects/esoxml/dtd/XmlStd.dtd">
---
> <!ENTITY % XmlStd SYSTEM "http://DUMMYHOST:DUMMYPORT/RETRIEVE?internal=XmlStd.dtd">
5c5
<            "http://www.eso.org/projects/esoxml/dtd/ngamsInternal.dtd">
---
>            "http://DUMMYHOST:DUMMYPORT/RETRIEVE?internal=ngamsInternal.dtd">

[rtobar]

Thanks @szampier, that looks all more or less fine.

I've ended up going through a bit of a rabbit hole and fixed a couple of things:

• The DOCTYPE declarations in the generated ngamsStatus XML documents in the server did not use the Host HTTP request header to populate the URL for the RETRIEVE command that would provide the ngamsStatus.dtd file, which could be an issue depending on the installation (e.g., if the server is behind a NAT, or depending on DNS configurations). I've fixed that.
• I added the ngamsInternal.dtd and XmlStd.dtd files to the list of allowed files to be retrieved via the RETRIEVE?internal= flag
• I'm now replacing those DUMMYHOST:DUMMYPORT strings with the correct server URL using the same logic that the one for the first point.

All of that provides now with the infrastructure to retrieve the DTD files... only to expect loads of validation failures, as initally expected. I'm using lxml to validate the output of a vanilla STATUS command. The first error is that the Status field is returning SUCCESS instead of OK/FAILURE/-, but then if I fix that I get that the NgamsStatus element is missing a Header sub-element

All of this makes me wonder what's the situation for the responses from the ESO servers? Our DTDs have clearly diverged from what we're serving, but it's not clear whether that happened before any of our changes or before we even adopted NGAS into our git repo (~11 years ago, from ALMA).

I've pushed the changes to the issue-103 branch anyway, as they shouldn't be harmful. If anything, they open the door to trying to align the schemas and the documents we produce.

[szampier]

Hi @rtobar , the last changes seem to have fixed the problem on our side. We will run some more tests until the end of the week, if you don't hear from us it means that it's all fine and you can merge the changes. Thanks a lot for the fix!

[szampier]

closing issue after merging PR #104