Issue accessing FAIR Data Point after first deployment about fairdatapoint

[kburger]

The default users are [email protected] (admin) and [email protected] (user), for both accounts the default password is password. Don’t forget to change the default user accounts as soon as your FAIR Data Point becomes publicly available.

[kburger]

Hi @vemonet, thanks for this detailed report. I have no experience in running the nginx base image with the VIRTUAL_HOST and LETSENCRYPT_HOST variables, so I don't know if that conflicts with any of the recommended setup in the Production Deployment docs. Would you mind taking a look at that page and check if your setup achieves a similar result?

Maybe @janslifka can provide additional input.

[vemonet]

Thanks for the help!

I logged in with [email protected], but still getting some pages showing 404 (the landing page and my-metadata):

Is it normal?

I was wondering also if the API is supposed to be exposed too or if everything is done through the web UI. The documentation does not talk about it, and I tried to expose it but couldn't reach it

Do you know if there is a specific path in the URL to access the API? (e.g. /api-docs)

[kburger]

If set up properly, the client acts as a gateway and proxies the traffic to the API through content negotiation. I have no experience in setting it up using the tools you've mentioned (nginx-proxy and the letsencrypt companion), so I cannot help you there. The procedure described in the production deployment docs always have worked for me, would it be possible for you to follow the steps described there and see if the data is served properly? The deployment at https://app.fairdatapoint.org should give you an idea on what it should look like.

The apidocs are visible on the /swagger-ui.html endpoint, which generates the swagger docs endpoint (https://fairdatapoint.semanticscience.org/v2/api-docs in your case).

[vemonet]

The nginx proxy is serving properly the web app now, it seems like the issue comes from the API
Here is the issue raised in the docker logs:

fdp_1 | 2020-11-12 13:34:05,621 269512 [http-nio-80-exec-4] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://fdp/spec
fdp_1 | 2020-11-12 13:34:05,621 269512 [http-nio-80-exec-5] INFO  nl.dtls.fairdatapoint.api.filter.LoggingFilter - http://fdp/meta
fdp_1 | 2020-11-12 13:34:05,631 269522 [http-nio-80-exec-5] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Original requesed url http://fdp/meta
fdp_1 | 2020-11-12 13:34:05,631 269522 [http-nio-80-exec-5] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Modified requesed url https://fairdatapoint.semanticscience.org/meta
fdp_1 | 2020-11-12 13:34:05,634 269525 [http-nio-80-exec-5] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Original requesed url http://fdp/meta
fdp_1 | 2020-11-12 13:34:05,634 269525 [http-nio-80-exec-5] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Modified requesed url https://fairdatapoint.semanticscience.org/meta
fdp_1 | 2020-11-12 13:34:05,636 269527 [http-nio-80-exec-5] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - 
    No metadata found for the uri 'https://fairdatapoint.semanticscience.org'

It says No metadata found for the uri 'https://fairdatapoint.semanticscience.org' when querying the API

Do you know if there are initialization steps we need to go through?

Note that I enabled CORS requests in my browser to make sure CORS request works

I exposed the API port 8080 on https://fdp-api.137.120.31.101.nip.io/swagger-ui.html

This URL redirects what is normally exposed on http://localhost:8080 of the fdp API container

[vemonet]

I have the exact same issue when doing the local deployment:

Error in Docker logs:

fdp_1         | 2020-11-12 13:56:50,263 82441 [http-nio-80-exec-1] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Original requesed url http://fdp/expanded
fdp_1         | 2020-11-12 13:56:50,264 82442 [http-nio-80-exec-1] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Modified requesed url http://localhost:8081
fdp_1         | 2020-11-12 13:56:50,265 82443 [http-nio-80-exec-1] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - 
    No metadata found for the uri 'http://localhost:8081'

And same for the API locally, when I expose the port 8080 I am not able to access the API (/swagger-ui.html)

My application.yml:

instance:
    clientUrl: http://localhost:8081
security:
    jwt:
        token:
            secret-key: SECRET
repository:
    type: 1

My docker-compose.yml:

version: '3'
services:
    fdp:
        image: fairdata/fairdatapoint:1.6.0
        volumes:
            - ./application.yml:/fdp/application.yml:ro
        ports:
            - 8080:8080

    fdp-client:
        image: fairdata/fairdatapoint-client:1.6.0
        ports:
            - 8081:80

    mongo:
        image: mongo:4.0.12
        volumes:
            - ./mongo/data:/data/db

    blazegraph:
        image: metaphacts/blazegraph-basic:2.2.0-20160908.003514-6
        volumes:
            - ./blazegraph:/blazegraph-data

If you rerun this on a clean local install you should face the same issues

[kburger]

The setup seems to be missing an environment value for the client service, as shown in the local deployment docs.

  environment:
    - FDP_HOST=fdp

Making the full docker-compose.yml:

version: '3'
services:
  fdp:
    image: fairdata/fairdatapoint:1.6.0
    volumes:
      - ./application.yml:/fdp/application.yml:ro

  fdp-client:
    image: fairdata/fairdatapoint-client:1.6.0
    ports:
      - 8081:80
    environment:
      - FDP_HOST=fdp

  mongo:
    image: mongo:4.0.12
    volumes:
      - ./mongo/data:/data/db

  blazegraph:
    image: metaphacts/blazegraph-basic:2.2.0-20160908.003514-6
    volumes:
      - ./blazegraph:/blazegraph-data

This setup works for me locally: the UI responds as expected, and the client proxies the requests to the server as expected:

$ curl -H "Accept: text/turtle" http://localhost:8081
...
Content-Type: text/turtle
...
<http://localhost:8081> a <http://www.re3data.org/schema/3-0#Repository>
...

[vemonet]

Hi @kburger the

    environment:
      - FDP_HOST=fdp

Got lost in translation but I retried with it and get the same error

I tried with exactly your docker-compose.yml but I am still getting the same issue

fdp_1         | 2020-11-13 09:27:21,285 23108 [http-nio-80-exec-7] INFO  org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:7}] to mongo:27017
fdp_1         | 2020-11-13 09:27:21,287 23110 [http-nio-80-exec-7] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Original requesed url http://fdp/expanded
fdp_1         | 2020-11-13 09:27:21,287 23110 [http-nio-80-exec-7] INFO  nl.dtls.fairdatapoint.util.HttpUtil - Modified requesed url http://localhost:8081
fdp-client_1  | 172.23.0.1 - - [13/Nov/2020:09:27:21 +0000] "GET /actuator/info HTTP/1.1" 200 101 "http://localhost:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0" "-"
fdp_1         | 2020-11-13 09:27:21,289 23112 [http-nio-80-exec-7] ERROR nl.dtls.fairdatapoint.api.controller.exception.ExceptionControllerAdvice - No metadata found for the uri 'http://localhost:8081'

Connection to mongodb and users works fine, the issue is that the triplestore under the API is not populated.

I am running on a completly new machine with a empty triplestore

I guess you are running in a test environment with a triplestore already initialized.

We can see that the FDP api (fdp_1) try to query for the URL of the app, but it cannot find metadata about it in the triplestore

So the triplestore should be initialized at start with the metadata FDP expect to find.

If I understand well here your client act like a UI and API? The Swagger UI should be accessed on the fdp-client then?

Some more info on my setup: we got the same issue on my laptop (Ubuntu 18.04) and production server CentOS 7

[kburger]

I'm running in a clean directory with your exact setup, from scratch with an empty triplestore. I understand why the triplestore is empty, the configuration is setup to use the in-memory store, so the blazegraph instance is not used. It's a bit of a legacy setup that needs rework, but the full set of options can be found here: Advanced Configuration: Triple Stores.

One important thing to note, and I realize now the documentation is lacking on this front, is that running a persistent setup (volumes mounted in docker and persisted on your disk) and changing the instance.clientUrl or instance.persistentUrl configuration values will result in undefined behaviour. We're working on a feature that allows for local to persistent id migration, but that's not done yet. So the current approach is to remove the mongo and blazegraph volumes when changing the configuration values.

The client acts as a gateway for the server too, so any interaction is intended to be with the client primarily (see the docs about an explanation). The client will then figure out based on the request to which underlying service it should go. So all API requests should be directed to the client address, the UI is served from the client address, and the swagger-ui is served from the client address.

Something that might go wrong if the underlying components are addressed directly, is some logic that bootstraps data at first interaction. Sometimes this might result in a wrong primary subject of the metadata, hence some of the "could not find metadata for x" errors.

Given the example on how to run locally with blazegraph (see the docs), I can run it locally and have everything working as intended. I'm running on OSX, but all of our server deployments run on ubuntu.

# application.yml

instance:
  clientUrl: http://localhost:8081
repository:
    type: 5
    blazegraph:
        url: http://blazegraph:8080/blazegraph

# docker-compose.yml

version: '3'
services:

    fdp:
        image: fairdata/fairdatapoint:1.6.0
        volumes:
            - ./application.yml:/fdp/application.yml:ro

    fdp-client:
        image: fairdata/fairdatapoint-client:1.6.0
        ports:
            - 8081:80
        environment:
            - FDP_HOST=fdp

    mongo:
        image: mongo:4.0.12
        volumes:
            - ./mongo/data:/data/db

    blazegraph:
        image: metaphacts/blazegraph-basic:2.2.0-20160908.003514-6
        ports:
            - 8080:8080
        volumes:
            - ./blazegraph:/blazegraph-data

Once up and running, you can either point your browser to http://localhost:8081, or use your favorite command line utility to retrieve the metadata:

curl -H "Accept: text/turtle" http://localhost:8081

Could you give this a go and let me know if this works on your machine?

[vemonet]

Thanks @kburger

I am using the exact same application.yml and docker-compose.yml

I managed to get it running! But still not seems to be ready for deployment

As you said the persistent storage seems to be the issue:

1. if I delete mongodb and blazegraph persistent volumes before running the docker-compose up then FDP will start without any issue!

sudo rm -rf mongo     
sudo rm -rf blazegraph

2. I stop the docker-compose up with ctrl+C
3. If I restart the docker-compose up : now I get 404 not found (exact same error as before)

This could be due to the initialization process that does not behave well when restart

Or the blazegraph image not working with persistent volume. I myself usually use this image: lyrasis/blazegraph:2.1.5 (volume shared on /data)

[kburger]

docker-compose up -d starts the services in daemon mode. Another docker-compose up should produce some conflicts, since the services are already running. The proper way of taking the services down is docker-compose down, then delete any persistent volumes, make any changes to the configuration, and then bring the services online again using docker-compose up -d.

I have no experience running that specific flavor of blazegraph, but as long as it is a regular blazegraph instance all should be fine I assume?

[vemonet]

Yes I know, but this means the data for Blazegraph is not persistent

Any Docker container running in production is expected to go down from time to time, which is fine because they will be started automatically and reuse the same data stored in the persistent volume (this is the main principle of containers and persistent volumes)

But with this FDP deployment if my docker container go down it will restart auto and will not be able to pick the persistent databases created at the first deployment (giving the 404 Not found error)

Meaning that everytime there will be the slightest problem with the Docker container I will need to manually reload all data I previously put in....

What is your process when you want to restart the docker-compose in production without losing all the FDP data? (maybe the FDP deployment works better with other triplestore in production)

So using the persistent volume in your FDP deployment seems counterproductive, because anyway your deployment is not gonna be able to reuse the persistent databases. So you better go with full ephemeral storage

[vemonet]

@kburger I am not running multiple docker-compose up

As I said: I am running it, then getting it down, then restarting it

And the issue is that during the second deployment, FDP is not able to read the blazegraph triplestore it created at the first deployment

You can reproduce by doing exactly:

docker-compose up -d
docker-compose down
docker-compose up -d

[vemonet]

The first docker-compose up -d starts a working FDP

The second docker-compose up -d starts a broken FDP which is not able to find his description in the triplestore.

[vemonet]

I am exploring using our public GraphDB triplestore as backend, but it seems like username and password cannot be defined for GraphDB triplestore: #96

[vemonet]

I fixed the issue with restarting the Blazegraph, it was related to permissions of my setup, thanks a lot for the help!

[xiaofengleo]

I fixed the issue with restarting the Blazegraph, it was related to permissions of my setup, thanks a lot for the help!

Hi @vemonet , I have the same issue with yours, I restart the Blazegraph, but the 404 still remains. Can you say a bit more about the permission? Thanks.