hifis-net / ansible-role-rsd Goto Github PK

See https://github.com/research-software-directory/RSD-as-a-service/tree/main/data-migration.

There are two new environment variables that enable matomo webstats in the RSD:

research-software-directory/RSD-as-a-service@99cff62

The next release of the RSD will introduce changes to the naming of the environment variables, see research-software-directory/RSD-as-a-service@c79841c

Check if requirements are met before setting up the RSD:

• Docker SDK for Python
• docker available
• docker-compose available
• ...

When using

    rsd_migrate_spotlights: true
    rsd_spotlight_migration_image: >-
      ghcr.io/hifis-net/rsd-spotlight-migration:latest

The ansible role crashes in the task Copy docker-compose.yml:

fatal: [*********.de]: FAILED! => {"changed": false, "checksum": 
"08f25fcdc74731da899da1c735b4599835bea5a3", "exit_status": 1, "msg": "failed to validate", "stderr":
 "The POSTGRES_AUTHENTICATOR_PASSWORD variable is not set. Defaulting to a blank string.\nThe 
POSTGRES_DB variable is not set. Defaulting to a blank string.\nThe Compose file 
'/home/******/.ansible/tmp/ansible-tmp-1666613486.006113-46971-4951698144570/source' is invalid 
because:\nUnsupported config option for services.initial-spotlights: 'pull_policy'\n", 
"stderr_lines": ["The POSTGRES_AUTHENTICATOR_PASSWORD variable is not set. Defaulting to a blank 
string.", "The POSTGRES_DB variable is not set. Defaulting to a blank string.", "The Compose file 
'/home/******/.ansible/tmp/ansible-tmp-1666613486.006113-46971-4951698144570/source' is invalid 
because:", "Unsupported config option for services.initial-spotlights: 'pull_policy'"], "stdout": 
"", "stdout_lines": []}

As far as I could see this is related to docker-compose not being the latest version 2.x:

terra-money/LocalTerra#46

Override default params for github-changelog-generator using .github_changelog_generator file.

See research-software-directory/RSD-as-a-service#672

• Debian 10 Buster
• Debian 11 Bullseye

Sometimes it is necessary to remove the previous RSD containers and/or volumes (for example if a new database table or column was added). It would be nice to have a way to remove the previously run containers. For database changes it is also needed to remove the connected volume rsd_pgdb, so a second option in the role for removing volumes would also be needed.

In this Ansible role project poetry is used in favour of pipenv.

See research-software-directory/RSD-as-a-service#430

See https://git.gfz-potsdam.de/id2/hifis/rsd/spotlight-migration

Currently, the certificate files have to be located in a directory with the same name as the inventory directory.

- name: "Copy TLS certificates"
  ansible.builtin.copy:
--> src: "{{ (inventory_dir | basename, item.file) | path_join }}"  <---
    dest: "{{ (item.dest, item.file | basename ) | path_join }}"
    owner: "root"
    group: "root"
    mode: "{{ item.mode }}"
  loop:
    - { file: "{{ rsd_tls_cert_file }}", dest: "/etc/ssl/certs", mode: "0644" }
    - { file: "{{ rsd_tls_key_file }}", dest: "/etc/ssl/private", mode: "0600" }
  no_log: True

It should be independent of the Inventory in the future.

The environment template file rsd-secrets.env.j2 is currently not used due to multiple issues:

• wrong file name stored in default role variable rsd_environment_file
• task "Copy environment file" requires the module ansible.builtin.template but is using the copy module instead

Manage citation metadata as part of the source code repository by providing a file codemeta.json (machine-readable information about your software including citation metadata) or providing a file CITATION.cff (human/machine-readable citation metadata).

If the image tag does not change, a new image is not pulled by community.docker.docker_compose. This happens on the latest image tag.

The expected behavior on running the role again with the latest tag is that the new image is pulled, but this is not the case.

Possible solutions:

• set pull: true in community.docker.docker_compose task, but that would always pull the image(s)
• introduce a new configuration variable to force pull
• set pull to true if rsd_version is latest

For now it is not possible to change the values for MAX_REQUESTS_GITHUB, MAX_REQUESTS_GITLAB, MAX_REQUESTS_DOI and RSD_ADMIN_EMAIL_LIST (recent new feature)

The support for Hotjar was removed with https://github.com/research-software-directory/RSD-as-a-service/pull/565/files

Currently, vfs is used as storage driver in Molecule tests If possible, this should be replaced by overlay2.

The reason for chosing vfs (copied from #11 (comment)): The Docker daemon did not start properly using overlay2, overlay, or devicemapper:

TASK [geerlingguy.docker : Ensure Docker is started and enabled at boot.] ******
fatal: [instance_rsd]: FAILED! => changed=false 
  msg: |-
    Unable to start service docker: Job for docker.service failed because the control process exited with error code.
    See "systemctl status docker.service" and "journalctl -xe" for details.

When using the default aufs storage driver, building Docker images failed:

TASK [rsd-role : Build containers with docker-compose] *************************
fatal: [instance_rsd]: FAILED! => changed=false 
  errors: []
  module_stderr: ''
  module_stdout: |-
    Step 1/11 : FROM node:14-alpine
     ---> 6f5dba13ae83
    Step 2/11 : RUN mkdir /app
  msg: 'Error: build failed with (<Service: admin>, ''error creating aufs mount to /var/lib/docker/aufs/mnt/1de234b3b477398643d1f2766c6e08a363a903a61dc7121eef9309ff16d8eef6-init: mount target=/var/lib/docker/aufs/mnt/1de234b3b477398643d1f2766c6e08a363a903a61dc7121eef9309ff16d8eef6-init data=br:/var/lib/docker/aufs/diff/1de234b3b477398643d1f2766c6e08a363a903a61dc7121eef9309ff16d8eef6-init=rw:/var/lib/docker/aufs/diff/aba21078893c364b22655c9e0e5670a996809bce3bfe38b010e582103227e2cc=ro+wh:/var/lib/docker/aufs/diff/39e9ed87202c43a70fd0cfffc6d5b2c21ced98cd16cfd10d69e354965c0095bb=ro+wh:/var/lib/docker/aufs/diff/8a680c493805ed7b28767cf21284d46cada253085b49b91d793c93750cbee25f=ro+wh:/var/lib/docker/aufs/diff/708cb20196eec28046307a58287351758e05d59e4396f15bb118ac79d72bbac0=ro+wh,dio,xino=/dev/shm/aufs.xino: invalid argument'')'

Unfortunately, the vfs driver was the only storage driver that worked (at least for me on my local machine).
The hint to use vfs did I found in this issue: geerlingguy/raspberry-pi-dramble#166

ORCID has been added as a login provider. For required changes, see research-software-directory/RSD-as-a-service@ede518e

With every release of the RSD, a migration script will be provided here:

https://github.com/research-software-directory/RSD-production/tree/main/database-migration/migration-scripts

It would be handy if we could run those automatically when deploying a new version. Please note that the versions specified in the file names refer to the RSD version, and not the version of the database image!

There exists a new environment variable ENABLE_OAIPMH_SCRAPER, see here.

See research-software-directory/RSD-as-a-service#678

This was introduced with RSD-as-a-Service v1.13.0.

To avoid data being lost during updates, it would be handy to automatically display a message when we are updating the RSD.

During the last update, I used a replacement nginx template to solve this issue manually:

    location / {
        # Allow quick enabling/disabling of maintenance messages
        if (-f /usr/share/nginx/html/under_maintenance.html) {
            return 503;
        }

        [....]

        error_page 503 /under_maintenance.html;
        location = /under_maintenance.html {
            root /usr/share/nginx/html;
            internal;
        }
    }

As maintenance page I used a simple template:

If the update finished without errors, the maintenance page should disappear again.

Documented in https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuring-dependabot-version-updates#enabling-dependabot-version-updates.

It would allow us to automatically update Poetry dependencies and GitHub Actions.

Similar to HOTJAR_ID the environment variables MATOMO_URL and MATOMO_ID need to be added as variables to the frontend container in docker-compose.yml.

So at https://github.com/hifis-net/ansible-role-rsd/blob/main/templates/docker-compose.yml.j2#L87 those need to be added:

      ...
      - RSD_AUTH_PROVIDERS
      - HOTJAR_ID
      - MATOMO_URL
      - MATOMO_ID
      - NEXT_PUBLIC_SURFCONEXT_CLIENT_ID
      ...

If I am correct, the nginx container currently stores the log files for 52 days. This is at least what I assume from looking at /etc/logrotate.d/nginx:

/var/log/nginx/*.log {
        daily
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        create 640 nginx adm
        sharedscripts
        postrotate
                if [ -f /var/run/nginx.pid ]; then
                        kill -USR1 `cat /var/run/nginx.pid`
                fi
        endscript
}

We need to specify that we can remove logs after X days.

The current version is deprecated. Active development moved to RSD-as-a-service.

Currently the role variable {{ rsd_postgres_password }} is used for both env vars POSTGRES_PASSWORD and POSTGRES_AUTHENTICATOR_PASSWORD.

See https://postgrest.org/en/stable/auth.html?highlight=authenticator#authentication-sequence for more details about the authenticator role.

Swagger has been added as a container to the RSD. For required changes, see research-software-directory/RSD-as-a-service@46b21df

For docker-compose.yml validation the role is using the docker-compose command which may not be available on systems where Docker Compose has been installed as a CLI plugin for Docker which needs to be called via docker compose (without the - dash).

TASK [external/hifis.rsd : Copy docker-compose.yml] ******************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => changed=false 
  checksum: f36df77fa7a513d26a5df7fa2c308711900b8c1e
  cmd: docker-compose -f /root/.ansible/tmp/ansible-tmp-1661158400.6968887-31446-78589091894946/source config -q
  msg: '[Errno 2] No such file or directory: b''docker-compose'''
  rc: 2
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

This is described here: https://github.com/github-changelog-generator/github-changelog-generator#migrating-from-a-manual-changelog

This allows a fully automated changelog generation.

The following environment variables will be removed with the next release of the RSD:

• AUTH_SURFCONEXT_TOKEN_URL
• AUTH_HELMHOLTZAAI_TOKEN_URL

Instead, they will be fetched upon each request.

If changing a value for an environment variable only some containers are restarted, but this results in an undefined docker-compose state or containers loose their connection between each other. I would suggest to use docker-compose down + docker-compose up for each deployment to avoid errors.

Example: I change the value for rsd_hgfaai_client_id, then the containers auth and frontend are restarted (because the community.docker.docker_compose: state: present knows what changed), but the nginx container lost the connection to the upstream: connect() failed (113: No route to host) while connecting to upstream, ... only a restart of all containers helped in this case

There is a new change in the RSD coming up: research-software-directory/RSD-as-a-service#506

If this gets merged we will need an option so that some additional docker mounts are made (via docker-compose.yml), in case of the HIFIS RSD like so:

  frontend:
    ...
    volumes:
      - ./deployment/helmholtz/styles:/app/public/styles
      - ./deployment/helmholtz/data:/app/public/data
      - ./deployment/helmholtz/images:/app/public/images

I would think of having some role variable like rsd_custom_theme which is unset by default. But if it is set then it is used for additional mounts:

  frontend:
    ...
    volumes:
      - ./deployment/{{ rsd_custom_theme }}/styles:/app/public/styles
      - ./deployment/{{ rsd_custom_theme }}/data:/app/public/data
      - ./deployment/{{ rsd_custom_theme }}/images:/app/public/images

I am preparing the merge of this new feature here: hifis-net/RSD-as-a-service#48

In the initial dry-run, it is expected that things are missing since there hasn't been a real deployment yet.

TASK [external/hifis.rsd : Assert that required TLS files are present] ***********************************************************************************************************************************************************
fatal: [rsd-staging.hzdr.de]: FAILED! => changed=false 
  assertion: tls_cert.stat.exists
  evaluated_to: false
  msg: TLS certificate and/or private key missing!