artefactual / deploy-pub Goto Github PK
View Code? Open in Web Editor NEWDeployment tools
License: Other
Deployment tools
License: Other
Ansible deployment -- any changes in https://github.com/artefactual/deploy-pub merged to master or changes required in https://github.com/artefactual-labs/ansible-archivematica-src merged to qa/1.7.x
There is an error trying to install bower, which is not a dependency of the 1.5 branch so it shouldn't be trying.
It's being executed as 'ubuntu' when the AtoM dir is owned by 'vagrant'. We should change the default 'atom_user' and 'atom_group' in the Vagrantfile.
I've tried to provision a few times now using vars-singlenode-qa.yml instead of vars-singlenode-1.6.yml, and I keep getting the same error.
TASK [artefactual.archivematica-src : Checkout archivematica-storage-service code] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/usr/bin/git checkout --track -b qa/0.x origin/qa/0.x", "failed": true, "msg": "Failed to checkout qa/0.x", "rc": 1, "stderr": "error: Your local changes to the following files would be overwritten by checkout:\n\t
storage_service/static/js/vendor/base64.js\nPlease, commit your changes or stash them before you can switch branches.\nAborting\n", "stderr_lines": ["error: Your local changes to the following files would be overwritten by checkout:", "\tstorage_service/static/js/vendor/
base64.js", "Please, commit your changes or stash them before you can switch branches.", "Aborting"], "stdout": "", "stdout_lines": []}
to retry, use: --limit @/vagrant/singlenode.retry
When I look at src/archivematica-storage-service, I see that branch stable/0.10.x is checked out even though archivematica_src_ss_version is set to "qa/0.x".
roles/artefactual.archivematica-src/tasks/ss-main.yml looks good. The version corresponds to the yml file and force is set to "yes".
Investigating...
As f80bf57, if someone tries to deploy Archiveamtica using the deploy-pub/playbooks/archivematica it's unclear for a newcomer if stable or qa branches are being deployed. This just beat @mcantelon which is a sign that we're doing a poor job explaining it.
I suggest to have two environments:
deploy-pub/playbooks/archivematica
deploy-pub/playbooks/archivematica-qa
Or we could do a better job in the README file explaning how to download the right branch of the AM role (e.g. qa/1.7.x branch targets AM 1.7 and so on) and the right vars file.
I was looking at installing Archivematica via Vagrant today, but I have a Windows workstation and Windows isn't supported as a control machine with Ansible. I suppose I could try using the Windows Subsystem for Linux, but that's not really supported by anyone either.
In any case, I think this could be the solution:
https://www.vagrantup.com/docs/provisioning/ansible_local.html
This has been used with Koha to allow Windows-based developers to use Vagrant as well.
I'm going to look at trying this out tomorrow, and then I'll send a pull request if I get it up and running.
I installed Archivematica version 1.7.0 onto a digital ocean VM (using the digital-ocean-install-example.rst)
Running the singlenode.yml playbook failed attempting to install dependencies for ElasticSearch:
TASK [artefactual.elasticsearch : Install dependencies] *******************************************************************************************************************
fatal: [am-local]: FAILED! => {"changed": false, "msg": "No package matching 'openjdk-7-jre-headless' is available"}
I updated the playbook to replace '7' with '8' in the following 2 lines:
elasticsearch_apt_java_package: "openjdk-7-jre-headless"
elasticsearch_java_home: "/usr/lib/jvm/java-7-openjdk-amd64"
I was then able to re-run the playbook and complete the installation successfully.
Based on instructions at https://github.com/artefactual-labs/archivematica-acceptance-tests, create ansible tasks/roles that install the AM acceptance tests and their dependencies:
Compare the following two files
The first contains the following but the second does not:
mysql_databases:
- name: "{{ archivematica_src_am_db_name }}"
collation: "utf8_general_ci"
encoding: "utf8"
mysql_users:
- name: "{{ archivematica_src_am_db_user }}"
pass: "{{ archivematica_src_am_db_password }}"
priv: "{{ archivematica_src_am_db_name }}.*:ALL,GRANT"
host: "{{ archivematica_src_am_db_host }}"
TASK [artefactual.archivematica-src : Run SS django database migrations] *********************************************************************************************************************************************
le \"/usr/share/archivematica/virtualenvs/archivematica-storage-service/lib/python3.9/site-packages/django/db/backends/base/base.py\", line 214, in check_database_version_supported\n raise NotSupportedError(\ndjango.db.utils.NotSupportedError: MySQL 8 or later is required (found 5.7.44).
This is an example: https://github.com/artefactual/deploy-pub/blob/master/playbooks/archivematica/vars-singlenode-qa.yml#L15-L16
As described in artefactual-labs/ansible-elasticsearch#5 and artefactual-labs/ansible-elasticsearch#3 the oracle java installer referenced in the lines above no longer exists.
These lines could be removed entirely from the vars-singlenode files, relying on the defaults defined in the ansible-elasticsearch role instead.
To ease the use of tools like https://github.com/appbaseio/dejavu
We install gearmand as part of the singlenode.yml playbook for centos, but we need also to add the line
OPTIONS="--listen=127.0.0.1 --port=4730 --queue-type=builtin"
to /etc/sysconfig/gearmand.
We can also start using the gearmand role available at https://github.com/artefactual-labs/ansible-gearman
You need to manually restart it currently to get it working:
sudo service nginx reload
Also I think this repo could use some version tags as a lot of things are changed in it and it's quite hard to install and older version. You need to explicitly lookup a certain commit where it was changed.
Needed to be able to use in a non-Vagrant environment.
For using these in a non-Vagrant VM, need to be able to configure a firewall.
Using master at 86d70f4 without any changes, vagrant up fails in the following task:
TASK [artefactual.archivematica-src : Generate special sampledata transfers] ***
fatal: [am-local]: FAILED! => {"changed": false, "cmd": "/home/vagrant/archivematica-sampledata/createtransfers.py create-variously-encoded-files", "msg": "[Errno 2] No such file or directory", "rc": 2}
Using master at 86d70f4 without any changes, vagrant up
fails in the following task:
TASK [artefactual.archivematica-src : Install front-end dependencies] **********
changed: [am-local] => (item=/vagrant/src/archivematica/src/dashboard/frontend/appraisal-tab)
failed: [am-local] (item=/vagrant/src/archivematica/src/dashboard/frontend/transfer-browser) => {"changed": true, "cmd": ["npm", "install"], "delta": "0:00:16.859546", "end": "2018-03-20 15:43:55.774615", "item": "/vagrant/src/archivematica/src/dashboard/frontend/transfer-browser", "msg": "non-zero return code", "rc": 1, "start": "2018-03-20 15:43:38.915069", "stderr": "npm WARN prepublish-on-install As of npm@5, `prepublish` scripts are deprecated.\nnpm WARN prepublish-on-install Use `prepare` for build steps and `prepublishOnly` for upload-only.\nnpm WARN prepublish-on-install See the deprecation note in `npm help scripts` for more information.\nsh: 1: webpack: not found\nnpm ERR! file sh\nnpm ERR! code ELIFECYCLE\nnpm ERR! errno ENOENT\nnpm ERR! syscall spawn\nnpm ERR! [email protected] prepublish: `webpack --progress --colors --entry ./app.js`\nnpm ERR! spawn ENOENT\nnpm ERR! \nnpm ERR! Failed at the [email protected] prepublish script.\nnpm ERR! This is probably not a problem with npm. There is likely additional logging output above.\n\nnpm ERR! A complete log of this run can be found in:\nnpm ERR! /var/lib/archivematica/.npm/_logs/2018-03-20T15_43_55_756Z-debug.log", "stderr_lines": ["npm WARN prepublish-on-install As of npm@5, `prepublish` scripts are deprecated.", "npm WARN prepublish-on-install Use `prepare` for build steps and `prepublishOnly` for upload-only.", "npm WARN prepublish-on-install See the deprecation note in `npm help scripts` for more information.", "sh: 1: webpack: not found", "npm ERR! file sh", "npm ERR! code ELIFECYCLE", "npm ERR! errno ENOENT", "npm ERR! syscall spawn", "npm ERR! [email protected] prepublish: `webpack --progress --colors --entry ./app.js`", "npm ERR! spawn ENOENT", "npm ERR! ", "npm ERR! Failed at the [email protected] prepublish script.", "npm ERR! This is probably not a problem with npm. There is likely additional logging output above.", "", "npm ERR! A complete log of this run can be found in:", "npm ERR! /var/lib/archivematica/.npm/_logs/2018-03-20T15_43_55_756Z-debug.log"], "stdout": "\n> [email protected] prepublish /vagrant/src/archivematica/src/dashboard/frontend/transfer-browser\n> webpack --progress --colors --entry ./app.js", "stdout_lines": ["", "> [email protected] prepublish /vagrant/src/archivematica/src/dashboard/frontend/transfer-browser", "> webpack --progress --colors --entry ./app.js"]}
We removed the database creation from the archivematica role at artefactual-labs/ansible-archivematica-src@fb87517 so now it needs to be created using the percona role , as we do in atom.
Some acceptance tests need to be able to scp
files from VM to local and therefore need to know the password for the ubuntu user created by the ubuntu/xenial64 box. However, this user has no password (that I can ascertain)—see https://bugs.launchpad.net/cloud-images/+bug/1569237. Solution would seem to be to specify the ubuntu user's password in the archivematica-xenial/Vagrantfile.
A new variable has been added to the AtoM role to determine the ES config used. See artefactual-labs/ansible-atom@72ddedb
AtoM Xenial should be installing 2.5 instead of 2.4 (the default)
Columbia University Library would like a way to install Archivematica without Elasticsearch
As described here: artefactual-labs/archivematica-acceptance-tests#23.
This is not meant to be the only way to deploy Archivematica, so the solution needs to allow a person deploying Archivematica to pick whether or not Elasticsearch is installed.
This will be one step towards being able to deploy a 'headless' Archivematica instance, where the dashboard is used only as a REST api, driven by requests from outside applications (such as the automation tools).
Trying to connect to the shared folder (when using "atom_devbox" set to true) as explained in here was giving me a lot of permission issues from a Linux OS host. Using Vagrant synced folders solved the issues for me.
I installed Archivematica 1.7 using digital-ocean-install-example.rst
I used Ubuntu 16.04. The instructions still work great, but I noticed a few details that would be helpful to add / correct:
$ ansible-playbook user.yml
I also had real troubles executing this deployment from either my Mac Air latop or iMac desktop. In both cases I had repeated problems with a particular dependency because of TLS errors, due to my Macs using an old version of OpenSSL.
- downloading role 'nodejs', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-nodejs/archive/4.1.1.tar.gz
[ERROR]: failed to download the file: Failed to validate the SSL certificate for github.com:443. Make sure your managed systems have a valid CA certificate installed. You can use
validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this platform: /etc/ssl/certs, /etc/ansible,
/usr/local/etc/openssl. The exception msg was: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590).
[WARNING]: - geerlingguy.nodejs was NOT installed successfully.
I tried to resolve this by updating OpenSSL but couldn't get it to work. In the end, I found it faster to simply set up another digital ocean droplet from which to run the ansible scripts. (I'm not sure whether any of this last issue is worth noting in the installation instructions, but I note it here just in case anyone else comes across this problem).
Since we remove the database creation from the ansible-archivematica-src role, the playbook is failing.
As the percona role was updated to run in redhat/centos deployment here, we should add it to the roles and may be, use the same playbook in Ubuntu and RH.
create a requirements-160.yml, vars-singlenode-160.yml and maybe a singlenode-160.yml, or else change singlenode.yml to run 160 stuff by default.
We have a working packer configuration to build atom boxes, we should move it here and build also archivematica vagrant boxes.
On the other side, the elasticsearch replication role available here, doesn't seem to fit with the repo, we should create a new git repo for it.
Currently, we sync the "current" folder (the one we use to run "vagrant up") using "/vagrant" as path in the virtual machine.
Changing this path to /home/local-sync (or any other name), would allow users to put their own transfers on the same folder than the playbooks, and process them, beacause /home is configured as the default transfer source in the storage service
This is perhaps more of a question, or a suggestion. When using deploy-pub locally with Vagrant I see that deployments are significantly faster when I enable SSH pipelining with:
[ssh_connection]
pipelining = True
Would it be a good idea to include that directive in the ansible.cfg
documents in this repository? It seems like a good default, officially supported by Ansible. Or would that be a conflict with other tools you use like mitogen?
We have changed the Ubuntu version check to install the Percona server in artefactual-labs/ansible-percona@16a44e5
After those changes it installs Percona 5.6 by default in Ubuntu Trusty if it's not changed in the playbook using the new mysql_version_major
and mysql_version_minor
variables.
When restarting the vagrant VM (vagrant halt -> vagrant up), archivematica services (archivematica-dashboard, archivematica-mcp-server, archivematica-mcp-client) require manual start up:
$ sudo start archivematica-storage-service
$ sudo start archivematica-mcp-server
$ sudo start archivematica-mcp-client
$ sudo start archivematica-dashboard
Seeing the following errors in the upstart log files:
Invalid value for chdir: /usr/lib/archivematica/storage-service
Error: can't chdir to '/usr/lib/archivematica/storage-service'Invalid value for chdir: /usr/share/archivematica/dashboard
Error: can't chdir to '/usr/share/archivematica/dashboard'/proc/self/fd/9: 4: /proc/self/fd/9: /usr/lib/archivematica/MCPServer/archivematicaMCP.py: not found
/proc/self/fd/9: 7: /proc/self/fd/9: /usr/lib/archivematica/MCPClient/archivematicaClient.py: not found
Since we have two branches in Binder now, stable/0.8.x (using ES 1.x) and qa/0.9.x (using ES 5.x), which will require to use the official Ansible role from ES, and considering that Binder can only run in PHP 5.6 in both cases; I think it's better to create a new playbook for Binder using Ubuntu Trusty and the ES Ansible role or the ES Artefactual role based on the version.
Inspired by @jhsimpson in artefactual-labs/ansible-archivematica-src#191:
Now that there is an official v1.7.0 tag in Archivematica, we need a way to make sure that this repo has an easy way to install that version.
This could be done with a stable/1.7.x branch in this repo, and a v1.7.0 tag?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.