CORS problem when serving posix files on localhost about hsds

[jjaraalm]

Ah, yes that makes sense. I allow default read permissions on all domains. It looks like ACL checks are being done on OPTION requests which are always anonymous. aiohttp-cors does look to be the simplest solution.

[loichuder]

Following up on this, I tried to set up CORS with the module aiohttp-cors, at least for the root of the domain (see loichuder@da80064).

The OPTIONS request passes smoothly but the GET request leads to an AssertionError thrown by aiohttp_cors in the service node (the same already reported in aio-libs/aiohttp-cors#193)...

[loichuder]

After some digging, I saw that the CORS was in fact already manually handled in HSDS in the jsonResponse function. Putting aiohttp-cors on the top of that created a conflict which led to the error mentionned in my previous comment (#28 (comment)).

This being said, the issue still holds when using the HSDS jsonReponse CORS handling. I managed to solve it by removing the CORS in jsonResponse and using aiohttp-cors instead (see loichuder@39b13b9).

@jreadey : what do you think ? Would it be better to use aiohttp-cors then or am I in a special case?

[jreadey]

@loichuder - cool that you have the posix setup running!

Why was CORS failing (before you setup aiohttp-cors)? I thought the headers the jsonRespnse sets should be adequate.

I wasn't aware of the aiohttp-cors package. Does it add something that would be useful? I'm not oppose to using aiohttp-cors if that would be better (and you are willing to do a PR!).

In any case, it would be nice if the integration tests had a check for CORS. I usually work with the Python client, so may miss if there's a CORS regression.

[loichuder]

I don't know why CORS was failing although headers were apparently set adequately as you said.
Perhaps this would need additional investigations (only tested with Firefox) but I guess there are two factors here:

• I am running everything on localhost.
• I am not using nginx which also implements its own CORS from what I saw from the config.

As for aiohttp-cors, it is only adding CORS support for aiohttp. To me, the advantage is that it is more modular and flexible than having the headers set in jsonResponse (also it means that OPTIONS requests need not to be manually implemented). Last but not least, it solved my problem :)

[jreadey]

@loichuder - would you mind submitting a PR with your aiohttp-cors setup? Also it would be handy to have an integ test that verifies the cors response headers. I'll do some adhoc testing on my side.

BTW, you might have noticed I took the nginx container out for docker-compose.yml and docker-compose.posix.yml. It's been a bit problematic and I figure anyone whose looking at handling a large number of clients would want to use kubernetes anyway (k8s has it's own internal LB).

[jjaraalm]

Just to chime in, I reproduced a CORS issue on chrome, but it was due to trying to set specific headers. In my case it was X-Hdf-domain that was causing problems (since Host is not settable with fetch()). Using request options instead of headers solved the problem.

[jreadey]

Ah - so custom headers seem a bit problematic with CORS. Fortunately looks like the only one we use is X-Hdf-domain and that can be replaced by using the domain param.

@loichuder - where you using X-Hdf-domain?

[loichuder]

@loichuder - would you mind submitting a PR with your aiohttp-cors setup? Also it would be handy to have an integ test that verifies the cors response headers. I'll do some adhoc testing on my side.

No problem. I can do that.

BTW, you might have noticed I took the nginx container out for docker-compose.yml and docker-compose.posix.yml. It's been a bit problematic and I figure anyone whose looking at handling a large number of clients would want to use kubernetes anyway (k8s has it's own internal LB).

Yeah I noticed. In fact, in my early tests of POSIX, I removed it myself as it was easier for testing.

@jjaraalm Thanks for your input ! Well, I didn't have any fancy headers and I passed the domain as a parameter so no use of X-Hdf-domain as far as I know. I can check again but as I said, I feel that running everything in local might be the issue.

[jjaraalm]

Interesting, it aiohttp-cors fixes it, that's good, but it seems mysterious. I am also running every thing on localhost while working at home, both hsds and a development version of a web app. I tested both chrome and firefox without problems. This is what a request looks like on chrome:

GET /?domain=/shots/0 HTTP/1.1
Host: localhost
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36
Sec-Fetch-Dest: empty
authorization: Basic YWRtaW46YWRtaW4=
Accept: */*
Origin: http://localhost:3000
Sec-Fetch-Site: same-site
Sec-Fetch-Mode: cors
Referer: http://localhost:3000/
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9

and response headers

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, DELETE, PUT, OPTIONS
Access-Control-Allow-Headers: Content-Type, api_key, Authorization
Content-Type: application/json; charset=utf-8
Content-Length: 836
Date: Tue, 24 Mar 2020 14:19:55 GMT
Server: Python/3.7 aiohttp/3.6.2

[loichuder]

What do you use as authorization credentials ?

[jjaraalm]

For my test server, I'm using password auth with the defualt user/passwords (admin, test_user1, test_user2) etc...

My js calls look like:

  async GET(request) {
    const url = `${this.endpoint}${request}?domain=${this.domain}`;
    const response =  await fetch(url, {
      method: 'GET',
      mode: 'cors',
      headers: this.Headers,
      cache: 'no-cache'
    });
    if (!response.ok) {
      throw new Error(`HSDS Error ${response.status} : ${response.statusText}`);
    }
    return await response.json();
  }

where headers are

    this.Headers = new Headers();
    this.Headers.append("Authorization",
                        `Basic ${base64.encode(`${username}:${password}`)}`);

I've only tested in fetch()-capable browsers and haven't tested polyfills for IE yet.

[loichuder]

Thanks, I will give a try with your method. I was using axios to fire the requests so that might be misconfiguration from my side.

[loichuder]

Ok so I tried using the same js calls as you (with fetch) and it still doesn't work.

I tried to access /?domain=/home/test_user1/test/tall.h5 but my preflight request OPTIONS returns a 403 Forbidden with the following error messages:

• on Firefox:
  Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost/?domain=/home/test_user1/test/tall.h5. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)

• on Chrome:
  Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

It seems to me that my OPTIONS request is not handled by the part of HSDS that adds the CORS headers (jsonReponse).

[jjaraalm]

Hmm, I don't understand. Can you post the actual request/response?

Check your credentials and that your user actually has permission to access that domain. If I try to access with bad credentials, I get a 200 on the preflight, 401 on the response, but the same error message as you in the console. I don't get a 403 though... It looks like Access-Control-Allow-Origin is not being set on 401 and is inadvertently flagged as a CORS violation.

[loichuder]

Here are the request headers:

Host: localhost
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Access-Control-Request-Method: GET
Access-Control-Request-Headers: authorization
Referer: http://localhost:3000/
Origin: http://localhost:3000
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache

Tell me if you need something more as I would need to set up once again the version of HSDS without my CORS handling.

About credentials, there are working with no problem elsewhere (including with aiohttp-cors). However, there is indeed something weird: when looking at the logs of the service node, it appears that the user detected in preflight request is default (and not test_user1) who has indeed not the rights to access this domain. Could this be the cause ?

[jreadey]

Hey, I merged the aiohttp-cors PR (#32) to master but somehow the CORS headers are not getting added now (Python 3.8 incompat?).

I manually added the headers as a tmp hack to get the travis tests to pass: cdd4524.

@loichuder - Could you take a look and see what's up with aiohttp-cors?

[loichuder]

Yes, aiohttp-cors appears not to be Python3.8-ready yet. I will try to get a minimal example working in Python 3.8 to pinpoint the issues.

[loichuder]

Ok I have looked into this and here is my progress so far:

1. I managed to have a MWE for aiohttp_cors in Python 3.8 so that may not be the issue.
2. The failing tests were on the requests /about and /info. Correct me if I am wrong, but I think the latter is implemented by the head_node. Which means that aiohttp_cors should also be set up for the head_node.
3. Even with aiohttp_cors set up for every node, I cannot get CORS headers in responses from GET requests made by the requests module of Python. This might be because non-browser requests are handled differently by CORS as CORS headers are present in the responses from browser requests. I could obtain the CORS headers by simulating the preflight request (OPTIONS) by adapting this SO thread
4. As a aside, apparently, you cannot use Access-Control-Allow-Origin: * if Allow-Credentials is set to true (here is a link to SO but I could find others). I don't know yet if this comes into play.

To sum up, I would propose to:

• Implement CORS with aiohttp_cors for the head node
• Change the failing tests by firing an OPTIONS request instead of a GET.

[jreadey]

The /about and /info requests go to an SN node. The head node would typically not be accessed externally, so don't think we don't need to deal with CORS on the head nodde right now.

Using Options rather than GET in the test sounds good.

From the SO, it seems like Access-Control-Allow-Origin needs to be a config rather than a hard-coded '*' if authentication is used. I can do this.

What about binary responses? e.g. GET /datasets//value. Do they get CORS?

[jreadey]

I've checked in code to have CORS_DOMAIN be a config option (defaults as '*'): 3bfce39.

[loichuder]

The /about and /info requests go to an SN node. The head node would typically not be accessed externally, so don't think we don't need to deal with CORS on the head nodde right now.

All right. I misunderstood then.

Using Options rather than GET in the test sounds good.

Well, that is the only way I found to get CORS headers with requests from the Python requests package. I will implement this in another PR.

What about binary responses? e.g. GET /datasets//value. Do they get CORS?

Again, not when doing them with requests but they get CORS when looking at requests made from the browser.

[loichuder]

Tests implemented in #38.