islandora-devops / isle-dc Goto Github PK

[STUB] Please build this out for steps for user testing

• add a .sql script that runs upon the first mariadb init which will create the following:
  • drupal db along with drupal_db_user and password. (Default already)
  • fcrepo db along with fcrepo_db_user and password.
  • gemini db along with gemini db user and password.

Atm we have all the site urls follow this pattern SERVICE_NAME.COMPOSER_PROJECT_NAME.localhost however this causes issues when the site is hosted on a machine other than localhost for Chrome, as Chrome does a bit of magic for localhost urls that the other browsers don't do. Likely the only solution here is to drop .localhost from our urls and just have SERVICE_NAME.COMPOSER_PROJECT_NAME as the urls by default.

When I run make in the development branch, make runs and generates the expected files. When I run it in the master branch (commit e48f215), this happens:

mark@mark-ThinkPad-X1-Carbon-6th:~/Documents/hacking/isle-dc$ make
make: *** No targets specified and no makefile found.  Stop.

Any suggestions?

@birkland https://github.com/Islandora-Devops/isle-dc/blob/development/docker-compose.yml#L243 appears we are still using a private image for the alpaca connector?

Should we be building from https://github.com/Islandora-Devops/isle-alpaca/blob/dev/Dockerfile and pushing to the official Islandora Dockerhub now? I can then update the docker-compose.yml accordingly.

(would love to also assign you here but still can't seem to, are you a member yet? I can send an invite again)

I can do this but just want to be sure we're on the right track.

When running:

docker-compose -p islandora down \
                --rmi local \
                --volumes \
                --remove-orphans

or

docker-compose -p islandora up --remove-orphans --detach

Is there any issue to removed unneeded settings from the main docker-compose.yml?

Advise on how to incorporate elements / services/ config changes from recent Islandora upgrade from 1.0.0 to 1.1.0 https://gist.github.com/dannylamb/fc910f4d1730f21191f384d533502e24

This should wait until we have the full stack working again. (4.30.2020) per Weekly meeting standup.

• The MVP2a scripts/islandora/install-islandora.sh script needs to be merged with MVP2b scripts/drupal/install.sh as JWT, migrations etc are missing?

Additionally @nikathone has some good thoughts and ideas about work using the drupal image entry-point.sh which he can either open up a new ticket for or add here for further discussion as needed.

If you don't know to change docker-compose.drupal.mariadb.yml this will bite you. There is no documentation pointing out this file needs to be change but is needed if assigning an IP or domain.

mariadb-${COMPOSE_PROJECT_NAME-isle-dc}.${DRUPAL_SITE_HOST-traefik.me}


mariadb-${COMPOSE_PROJECT_NAME-isle-dc}-${DRUPAL_SITE_HOST-traefik.me}
---------------------------------------^

• Sample .env - Clean up .env and change to sample.env instead
• add to .gitignore
• Write up steps for how to do this in README

From @marksandford on Slack:

"I've gotten the test ingest files ready to go and zipped up. Instructions are in the Readme, but it's basically installed as a module and a series of drush commands are run.

One caveat is that I consistently get SQLSTATE[40001]: Serialization failure when running the media migration. Running it again with --update seems to fix it (as described in Readme). Setting mysql to read-committed avoids the problem entirely, but I have no idea what the implications are for that in production, nor do I know if the ISLE config will differ in ways that will avoid the error"

@dannylamb Have you run into this issue yet?

Move fcrepo and fcrepo related microservices to separate docker-compose.yml includes

• gemini
• recast
• milliner
• blazegraph

This allows people to not have to use it when setting up for the first time. Please note the fcrepo & gemini dbs will still be created if only using Drupal.

Create gemini, milliner and recast PRs - Danny (for Monday if possible)

• Update the https://github.com/Islandora-Devops/isle-dc/ with the appropriate services as needed

• Review and update code from Gavin on the following new fcrepo related microservices, issue PRs as needed.

• https://github.com/Islandora-Devops/isle-gemini
• https://github.com/Islandora-Devops/isle-milliner
• https://github.com/Islandora-Devops/isle-recast

Local Environment

This environment is intended for local development. Users will create a Drupal site in the folder codebase, which gets bind mounted into the drupal service container. Allowing developers to use composer / drush locally to work out of the codebase folder.

CODEBASE links to https://github.com/Islandora-Devops/isle-dc/blob/development/codebase

• Review Makefile again once we get build working with MVP3 and 2 merged.

• Ensure that value of docker_compose_project ?= can be changed from hard-coded islandora to match .env value instead otherwise it appears Docker volumes and container names don't match and cause issues

• Review @nigelgbanks 's suggestion here #6 (comment) re Minor non-issue. Should be added to .PHONY targets.

• Update README or documentation with syntax in Make file, benefits of using and scripts / actions it assumes.

• Once working remove wodby tags etc from docker-compose and envs.
  

• Put in default values in docker-compose and allow users to override

Currently the official Tomcat base image doesn't come with a ROOT.war or tomcat-admin panel. Is this wanted or needed?

From PR #16

Add this to the README.md

Depends on https://github.com/Islandora-Devops/isle-alpaca/pull/3

Comment in the build sections for and point to ^^ once it's merged (or pull it in and point to a local copy on disk).

## How to Test

From your isle-dc folder:
- `docker-compose down -v`
- `rm -rf codebase && mkdir codebase` to clean the codebase
- If you haven't already done so, `(cd scripts; ./generate_keys.sh)` to generate keys
- `docker-compose up --build -d`
-  `docker exec -it isle_dc_proto_solr bash -c "solr create_core -c islandora"`
- `docker exec -it isle_dc_proto_php bash -c "sh /scripts/islandora/install-islandora.sh"`

You can test by ingesting a repository item, giving it a media and tagging that media as an original file.  This should trigger derivatives.

Admittedly, there's still something super wonky with karaf/camel uploading derivatives, and sometimes you get uploaded files of 0.  I'm still working through that, but you can at least see that the whole system is functioning, albeit a bit impaired.

• Write up new README for how to use this build along with paths for materials for Power users
• For now avoiding using Make file due to oddities in testing, this works so far manually:

cp sample.env to .env
./scripts/drupal/init.sh --codebase islandora
docker-compose pull
docker-compose up -d
docker-compose exec drupal drush uli (once everything is done)

• Cleanup and remove old readmes accordingly

• For the future once the Make file is back in play update readmes accordingly

• BONUS: Update with all involved repos especially the https://github.com/Born-Digital-US/drupal-project

MVP3 Blazegraph needs an install script to be added to the soon to be merged in Makefile for this service to start up.

From the April 23 weekly call, @nigelgbanks raised the idea of discussing potentially using s6-overlay for

• @dbernstein seconded this for usage pushing log data in fcrepo to a logging service like Cloudtrail.
• @g7morris pointed out that the ISLE 7 project uses this service extensively
  • https://github.com/Islandora-Collaboration-Group/isle-apache/blob/master/Dockerfile#L41

The task should be to identify which images and services could benefit from utilizing s6-overlay to help with process management especially child processes handling as well as piping logs to stdout (as needed)

Also reference this issue: Logging to stderr / stdout for all images / containers / services #45

Drush 9.0+ is in the "Requirements" list. This is on the host where you're running docker-compose?

I've been running a local setup on my Mac OSX (Catalina) and running into extremely long page load times. The Manage->Extend page takes just shy of 30 seconds to load. Many pages take at least 4 seconds to load. It's a pretty awful developer experience.

@birkland researched adding the delegated flag to the volume mount for the website codebase. It seems promising and I'll submit a PR for that in a few minutes for other folks to try out.

After importing an object via migration_islandora_csv I attempted to edit an object. After doing so I get an empty manifest and a 500 error on the object's page.

Manifest

{
  "@type": "sc:Manifest",
  "@id": "http://islandora-isle-dc-12-34-56-78.traefik.me/node/839/manifest",
  "label": "IIIF Manifest",
  "@context": "http://iiif.io/api/presentation/2/context.json",
  "sequences": [{
    "@context": "http://iiif.io/api/presentation/2/context.json",
    "@id": "http://islandora-isle-dc-12-34-56-78.traefik.me/node/839/sequence/normal",
    "@type": "sc:Sequence"
  }]
}

Something interesting I saw in watchdog. 2 calls but one called the traefik.me URL and the other localhost.

openseadragon Request for IIIF manifest at https://islandora-isle-dc-12-34-56-78.traefik.me/node/840/manifest returned 0
openseadragon Request for IIIF manifest at http://localhost:8000/node/840/manifest returned 0

To export all settings the sync folder needs to be set correctly. Ideally this should be set by default

To fix

$ chmod 777 -R /var/www/drupal/config/sync

Hi,

The company I work for would like to start working with ISLE 8 ASAP. Is there some way we can help speed up the development? Is there a backlog of work? Can we do a small conference call to talk things through?

Ik would like to hear from you!

Thanks!

MVP3 - Accept latest alpaca PR islandora-deprecated/isle-alpaca#3

[STUB] Please build this out for steps for user testing

• Include an updated csv to this effect for migrations. This is meant to test tooling only not metadata conversion or concerns.

From #6 (review)

One minor fix for the reversion/

"@nikathone This looks good except on line 41 of the docker-compose.yml, should this section of the drupal service that is currently:

depends_on:
    drupal_db:
       condition: service_healthy

actually be changed to:

depends_on:
    mariadb:
         condition: service_healthy"

• Removes container name values in docker-compose
• Updates solr image with official 8.4.1.
• Adds a few .env values as needed
• Attempts to bind mount config.zip for users to have a unique schema.xml

[STUB] Please build this out for steps for user testing

Define SSL usage process for LE, commercial and self-signed certs in non-local environments

Running make fails with error:

ERROR: for blazegraph  Cannot start service blazegraph: error while mounting volume '/var/lib/docker/volumes/islandora_blazegraph-data/_data': failed to mount local volume: mount /c/Users/Dev/projects/isle-dc/data/blazegraph/data:/var/lib/docker/volumes/islandora_blazegraph-data/_data, flags: 0x1000: no such file or directory
Encountered errors while bringing up the project.
make: *** [Makefile:21: up] Error 1

changing the volume type to default (rather than local) fixes the issue. As no other volumes have type local, it probably makes sense to bring the blazegraph mount in line line with the others.

[STUB] Please build this out for steps for user testing

Please note Cary Gordon is also interested in testing this and should be included.

Compared to an install via islandora-playbook, the composer.json file from
Born-Digital-US/drupal-project is missing a couple dependencies:

"solarium/solarium": "^5.1",
"symfony/event-dispatcher": "4.3.4 as 3.4.99",

For now, it seems the best solution is to update the Born-Digital-US/drupal-project.

After ingesting a large tiff open seadragon shows a 500 error.

Unable to open [object Object]: HTTP o attempting to load TileSource

First time visiting a site that implements an insecure https request will cause the media to load within Open Seadragon.

Quick solution: is to go directly to https://islandora-isle-dc.12-34-56-78.traefik.me/cantaloupe/iiif/2 and accept the warning. This should probably be addressed, there is no obvious/simple indicator to the developer/user as to why the failure is happening.

Users need to manually add the domain to the hosts file as before but only for windows users.

• Alpaca - Issues with alpaca might be solved with switching over to nginx
• Danny L to review once merged and attempt to test via new setup again with nginx

I think there is remaining work to add SYN and get it working in the isle-fcrepo image?

@dannylamb Apologies if you've already done this but I don't see any commits per se?

https://github.com/Islandora-Devops/isle-fcrepo

Toronto is still coming up as default timezone not UTC for the newly created Drupal site when resetting admin password.

If you checkout an existing Drupal site into codebase and do make, the build will fail with

Composer could not find a composer.json file in /app
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section
make: *** [Makefile:15: drupal_init] Error 1

Suggestion: docker-compose.override.yml to set passwords and other options. Have MAKE file generate it to give a simplified file to use.

https://devilbox.readthedocs.io/en/latest/configuration-files/docker-compose-override-yml.html

Please feel free to close if this isn't a good idea.

Docker-compose is ignoring values set during the makemake/docker-compose up process. Not sure any other values are being ignored.

docker-compose.env.yml & docker-compose.yml have these values prior and after the make/docker-compose up process.

DRUPAL_DEFAULT_ACCOUNT_EMAIL: [email protected]
DRUPAL_DEFAULT_ACCOUNT_PASSWORD: abc1234567890

Observed the admin user has "[email protected]" with "password" as its password.

Merge MVP2a, 2b & 3 #6
Gavin to resolve merge conflicts in PR

• Rebuild and push all images to official Islandora Dockerhub repo
• Update all docker-compose information with new locations for images

@dannylamb Should we rebuild and clear this prior to merge work?

islandora-deprecated/isle-hypercube#7

Per @nikathone's comment here #6

"The drupal.Dockerfile still using registry.gitlab.com/nikathone/drupal-docker-good-defaults/php-nginx:latest for php and nginx base image but this will change once we have a CI to build images/nginx-php/Dockerfile."

@nikathone Can we make an official Islandora version of this post-merge? We now have control over the Islandora Dockerhub account to be able to push images. We can also make a new repo accordingly if need be too. Your thoughts?

Update activemq with configuration fix for login

From #21 (comment)

Re-testing the current setup has revealed now that the current wodby image and the Drupal build are having the following issues.

• Drupal Report Status now indicating You are using an incompatible Solr schema

• Solr path needs to be / not /solr.

This feels like something changed in the Drupal settings / code level. Putting back on the TO DO list for us to test / fix.

[STUB] Please build this out for steps for user testing