Reducing container startup time in non-export mode about ansible-galaxy-extras

[mvdbeek]

What about putting a empty file into /export during build-time.

this is what we're doing in https://github.com/ARTbio/GalaxyKickStart/blob/master/roles/galaxy.movedata/tasks/main.yml. As an added bonus you will be able to commit your docker images :).

[bgruening]

So you ended up with the same idea and named this file .original? :)

[mvdbeek]

exactly ... seemed like the easiest way to control the persistence, and works with containers and VMs.

[bgruening]

What are you plans for this sub-role? Why is this not upstream ;)

[mvdbeek]

Sure, let's do that. I'll make this a separate repo, and make it configurable if you want hard-links or soft links, and you'll use that for docker-galaxy?

[mvdbeek]

Hmm, though it would be a bit more work for the custom locations you're using. What we do is just take the location in the container (say /var/lib/postgresql-9.3/ and move that to /export/var/lib/postgresql-9.3/). Not a crazy amount of work, but that will need some testing.

[bgruening]

I guess soft-links are enough? If you have already ported the export_user_files.py.j2 to ansible than lets use this. I'm not entirely convinced that ansible is more readable than a python script but I'm willing to learn :)

I don't see a problem with the custom locations as these can be migrated if needed like here: https://github.com/galaxyproject/ansible-galaxy-extras/blob/master/templates/startup.sh.j2#L3

Wenn wir es uns richtig dreckig geben wollen könnten wir in dem Rutsch auch dem Umstieg auf Ubuntu 16.04 machen. With this change people need to upgrade there DB not matter what and we can introduce this the new location as well.

[mvdbeek]

I guess soft-links are enough?

For containers definitely, but for the VM use case you may want to upgrade the instance by replaying the playbook. Unfortunately ansible doesn't play nice with softlinks. Though in reality, you would probably just point a new VM against old exported data, just as in docker.
I'll make it configurable, maybe we can even remove the hardlink option. The downside with hardlinks is that with GalaxyKickStart containers we need to run privileged containers to be able to do bindmounts, so here soft-links would definitely be better.