Add support for installation in Bioconda about slim

[bhaller]

I've pinged two Messer Lab people who I think use conda, for initial testing. If that goes well I'll announce it on slim-discuss, which will reach ~100 people, for further early adopters. And if that goes well then I'll add a mention of it in the SLiM manual, which will eventually reach everyone. Thanks for making this happen. Stay tuned.

[bhaller]

A PhD student in our lab, Ian, has tested the conda installation of SLiM and said it went smoothly and performed identically to the standard install, so all seems to be well. I plan to send out an announcement on slim-announce and slim-discuss tomorrow evening, and it will be discussed in the SLiM manual in the next release of SLiM (whenever that happens).

[molpopgen]

@bhaller -- Happy to do a first deposit to bioconda. Not sure I can keep up with later releases. I'm already maintaining several recipes, and @jeromekelleher and I have to deal with an issue that is preventing msprime from being used as a dependency right now (annoying!!), which will likely affect the chances of getting the slim Python package into bioconda.

You will see that the recipes are rather simple. libsequence is similar to slim in that it has no external dependencies beyond a compiler. The files are here, and I basically just hacked those to make one for slim, but ran into the "make install" issue, which is why there's no PR.

The only real difference for slim is to change build.sh to use cmake and to remove skipping the building on os x.

[petrelharp]

I've added an install target to cmake, and updated the README that explains it... does that give you enough to go on, @molpopgen?

[molpopgen]

OK, thanks @molpopgen https://github.com/molpopgen and @petrelharp https://github.com/petrelharp. If this needs to go into a new release to make further progress, then it will roll in SLiM 3.2.1 or SLiM 3.3 (whichever it turns out to be), which I don't currently have any concrete plans for. So it may take a little while (a couple of months); but I guess there is no particular hurry on this? I don't want to roll SLiM 3.2.1 just for this addition.
Perhaps we can get the bioconda stuff ready, while we remember how to do it?

That's not a worry--it takes me about 10 minutes for cases like this. I've done a fair number of recipes already.

@molpopgen https://github.com/molpopgen, thanks for the tip re: -flto. I think Xcode is already doing this when one builds on OS X, but I'm not certain. For Linux etc. it would need to be added to the cmake stuff. @petrelharp https://github.com/petrelharp, if you already know how to add that, that would be swell, otherwise I can delve into cmake to try to figure out out. :-> I'll double-check that it is happening in Xcode already. I don't know how to do this already (and have to make slides before I do
anything else...)

For future record:
https://cmake.org/cmake/help/latest/module/CheckCXXCompilerFlag.html

Logically, it the above passes for -flto, then it must also be a linker flag, else the system is globally bonkers.

If I get a minute, I'll give a whirl shortly.

[jeromekelleher]

Just to weigh in my 10c here: a SLiM bioconda recipe would be great, and I'm happy to help if I can. Once the initial recipe is done, a new release is a one-line change to the recipe 99% of the t ime, so shouldn't be much of a maintenance burden.

[jeromekelleher]

It means that bioconda depends on conda-forge for lots of stuff, and anything that's in conda-forge should also be available to bioconda users.

[bhaller]

OK, so then there would be no need to put SLiM in bioconda; it would be available anyway already through conda-forge. That does sound good, then.

[jeromekelleher]

The initial recipe has been merged into conda-forge, so the slim-feedstock recipe should get created soon and the slim packages for linux/osx appear on anaconda after that.

Anyone fancy testing them out, perhaps doing a little perf comparison to see if this build is significantly different to a native one?

[jeromekelleher]

Here's the feedstock repo: https://github.com/conda-forge/slim-feedstock

And the anaconda page: https://anaconda.org/conda-forge/slim

Hmm, the package files are pretty corpulent at 2.5Mb. It'd be good to see what's in there and if it could be SLiMmed down (I'm here all week folks!).

[jeromekelleher]

OK, groovy. So if I understand correctly, this means that SLiM is now publicly available as an installable package through conda? I don't really know what the difference is between conda, conda-forge, anaconda and Bioconda; what would be a good/correct announcement to make publicly regarding this, and what link(s) should I supply in that announcement?

Yes, SLiM can now be installed from conda. Someone who uses conda should probably write some documentation for end users. Here's what I wrote for tskit. I'd recommend getting some feedback from initial users before making a full announcement. Maybe performance is terrible or something.

(Also, I think the link you supplied above for the anaconda page is identical to the link you gave for the "feedstock repo".)

Thanks, I fixed the link.

[jeromekelleher]

When the next version of SLiM is released, hopefully somebody can walk me through how to revise this to work with the new release, and then after that hopefully I can be self-sufficient.

I'm very happy to do this @bhaller, just ping me on an issue over at https://github.com/conda-forge/slim-feedstock when the next release is out.

[molpopgen]

I was just going to email you and @petrelharp about this. I'm a member of that org, and have merge permissions.

Yes, each new release requires an update to the recipe. The old builds are preserved so that one may conda install -c bioconda slim==3.3 at any time in the future, given that things like compiler dependencies are properly pinned.

[molpopgen]

I just did a quick hack of a recipe, but there is a road block. It looks like the cmake system is not set up to generate install targets:

cmake -D CMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$HOME/tmp ../SLiM-3.2
make -j 6 > /dev/null
make install

The output is:

make: *** No rule to make target 'install'.  Stop.

That needs to be fixed before a working recipe can be generated.

[bhaller]

@jgallowa07, you set up the cmake stuff with Boyana, right? Any chance of you guys doing a pull request to provide a "make install" target for it?

@molpopgen, thanks for looking into this! Are you offering to manage the Bioconda submissions for SLiM? That would of course be greatly appreciated! :-> We do a release every couple of months, so it wouldn't be a big thing, but I personally find a lot of this very Unix-y stuff a bit mystifying. I'm used to working in my Xcode IDE. :->

[molpopgen]

#27 is confirmed to work on my system:

 make install
[ 10%] Built target tables
[ 57%] Built target gsl
[ 70%] Built target eidos
[ 71%] Built target kastore
[100%] Built target slim
Install the project...
-- Install configuration: "Release"
-- Installing: /home/molpopgen/tmp/bin/slim
-- Installing: /home/molpopgen/tmp/bin/eidos

What we need is a release with this PR merged, and then I can do more testing. I do not have experience with "out of place" builds on bioconda, so there may be some false starts, but I can probably deal with it. Since all of my stuff are either Python packages or use GNU autotools, builds are all "in-place", so that is what I'm used to.

[molpopgen]

Looks like I'll be able to an "out of place" build from within the SLiM source repo by making the build dir in there first. That'll solve some potential problems on the conda/docker side.

On a related note, there are certainly compile-time optimizations being left out, esp. on modern Linux machines, or any machine with a compiler supporting "link time optimization". Not sure if OS X's current Xcode supports it, but it does another optimization pass at link time. One benefit of this is that some missed opportunities for inlining can be rescued. You apply -flto as a compiler and as a linker flag. Since only newer compilers support it, you'd need cmake's macros to check that "compiler has flag foo" and "linker has flag foo" and, if both pass, add the options. The only negative is longer link time.

[bhaller]

OK, thanks @molpopgen and @petrelharp. If this needs to go into a new release to make further progress, then it will roll in SLiM 3.2.1 or SLiM 3.3 (whichever it turns out to be), which I don't currently have any concrete plans for. So it may take a little while (a couple of months); but I guess there is no particular hurry on this? I don't want to roll SLiM 3.2.1 just for this addition.

@molpopgen, thanks for the tip re: -flto. I think Xcode is already doing this when one builds on OS X, but I'm not certain. For Linux etc. it would need to be added to the cmake stuff. @petrelharp, if you already know how to add that, that would be swell, otherwise I can delve into cmake to try to figure out out. :-> I'll double-check that it is happening in Xcode already.

[petrelharp]

OK, thanks @molpopgen <https://github.com/molpopgen> and @petrelharp <https://github.com/petrelharp>. If this needs to go into a new release to make further progress, then it will roll in SLiM 3.2.1 or SLiM 3.3 (whichever it turns out to be), which I don't currently have any concrete plans for. So it may take a little while (a couple of months); but I guess there is no particular hurry on this? I don't want to roll SLiM 3.2.1 just for this addition.

Perhaps we can get the bioconda stuff ready, while we remember how to do it?

@molpopgen <https://github.com/molpopgen>, thanks for the tip re: -flto. I *think* Xcode is already doing this when one builds on OS X, but I'm not certain. For Linux etc. it would need to be added to the cmake stuff. @petrelharp <https://github.com/petrelharp>, if you already know how to add that, that would be swell, otherwise I can delve into cmake to try to figure out out. :-> I'll double-check that it is happening in Xcode already. I don't know how to do this already (and have to make slides before I do

anything else...)

[bhaller]

It sounds like it's simple enough that once we get the machinery working the first time around I can probably do later releases myself, which would be logistically simpler. I've just put in a request to be added to the bioconda contributors list.

[molpopgen]

It sounds like it's simple enough that once we get the machinery working the first time around I can probably do later releases myself, which would be logistically simpler. I've just put in a request to be added to the bioconda contributors list.

Sounds good. Once things are ready, I'll get an initial release in.

[bhaller]

Hi @molpopgen. I've just released SLiM 3.2.1 (release tag in git: v3.2.1), which adds in the make install support from @petrelharp. I think that means that it should now be possible to add SLiM in Bioconda, so if you could try setting up the initial release that would be great. :-> Thanks! @petrelharp, is pyslim already under Bioconda?

By the way, re: LTO, your fix for that just rolled out in 3.2.1 as well, thanks @molpopgen. I added it to the Xcode project too, for Release builds. I haven't done any testing to see whether it actually made anything appreciably faster; I'm waiting for a new desktop machine and that will be the first thing I do with it. I don't want to make my laptop that busy, and I don't really like running speed tests on the cluster because the results probably depend too much on what else is going on with the other cores...

[molpopgen]

Hi @molpopgen. I've just released SLiM 3.2.1 (release tag in git: v3.2.1), which adds in the make install support from @petrelharp. I think that means that it should now be possible to add SLiM in Bioconda, so if you could try setting up the initial release that would be great. :-> Thanks! @petrelharp, is pyslim already under Bioconda?

Sounds good. I'm bogged down until about Feb 18. I'll get to it then-ish. The timing is good, as there are a few recipes I need to update. I'd put it aside while they were rebuilding the world vs GCC 7 (finally!). IIRC, @jeromekelleher got msprime working under the new conda setup, so hopefully other projects can get updated, too.

Wouldn't pyslim have a dependency on slim? @petrelharp

By the way, re: LTO, your fix for that just rolled out in 3.2.1 as well, thanks @molpopgen. I added it to the Xcode project too, for Release builds. I haven't done any testing to see whether it actually made anything appreciably faster; I'm waiting for a new desktop machine and that will be the first thing I do with it. I don't want to make my laptop that busy, and I don't really like running speed tests on the cluster because the results probably depend too much on what else is going on with the other cores...

Hard to know what the effect is. I saw in your release notes that builds take longer w/LTO, which implies a lot of redundant symbol generation across object files. So the main result may be a smaller binary but you may also recover lost inlining, etc..

When I test on the cluster here, I request the entire node.

[bhaller]

OK, this all sounds good. No rush, Feb. 18-ish is fine.

When I test on the cluster here, I request the entire node.

Huh. I'm not sure whether that's something I can do on the Cornell cluster I use. When you do that, do you then utilize just a single core on the node? Or if you use all the cores, how do you prevent the same problem? I generally run my timing tests using just a single core, and try to keep the machine unoccupied otherwise, but that means they take a long time...

[molpopgen]

Huh. I'm not sure whether that's something I can do on the Cornell cluster I use. When you do that, do you then utilize just a single core on the node? Or if you use all the cores, how do you prevent the same problem? I generally run my timing tests using just a single core, and try to keep the machine unoccupied otherwise, but that means they take a long time...

I do a mix of things. Sometimes I run just one job, but I often completely load the node, as that is the more relevant benchmark, given how simulations are run in practice. For me, the most relevant thing right now is timing running 128 jobs running at once on our newest AMD nodes.

In fwdpy11, it is easy to automate multi-processor use via concurrent.futures, so I can test 1, 64, 128 simulations running at the same time, all from the same script.

[bhaller]

Hi @molpopgen. I'm starting to put together the next release of SLiM, perhaps by the end of the month, so it would be good to have this Bioconda stuff working. If you don't have time to deal with it, maybe @jeromekelleher or @petrelharp does? Thanks to whoever can field this; I just have no idea what to do with it, not being that Unix-y. :->

[molpopgen]

We can deal with it when a release is available.

…

On Fri, Apr 5, 2019, 7:43 AM Ben Haller ***@***.***> wrote: Hi @molpopgen <https://github.com/molpopgen>. I'm starting to put together the next release of SLiM, perhaps by the end of the month, so it would be good to have this Bioconda stuff working. If you don't have time to deal with it, maybe @jeromekelleher <https://github.com/jeromekelleher> or @petrelharp <https://github.com/petrelharp> does? Thanks to whoever can field this; I just have no idea. :-> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGHnH_93lx92XMRWqFg5V1oi5AWmHiviks5vd2EFgaJpZM4YYkpX> .

[bhaller]

@molpopgen, that has been done already; the support needed was added in SLiM 3.2.1 (see my comment above, Jan. 31). It would be good to try to get that release under Bioconda, so that if there is any issue with doing so, it can be resolved in the SLiM 3.3 release that is planned. So I think now is the time to do this.

[molpopgen]

If you want the current release, copy an existing recipe and go for it. I don't have the extra bandwidth right now, and you'll have to learn this stuff anyway, for the future, as the maintainer.

…

On Fri, Apr 5, 2019, 7:57 AM Ben Haller ***@***.***> wrote: @molpopgen <https://github.com/molpopgen>, that has been done already; the support needed was added in SLiM 3.2.1 (see my comment above, Jan. 31). It would be good to try to get that release under Bioconda, so that if there is any issue with doing so, it can be resolved in the SLiM 3.3 release that is planned. So I think now is the time to do this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGHnH-0TqBVTgJRv1B07YoXO8YJ-vbKFks5vd2QSgaJpZM4YYkpX> .

[bhaller]

Well, this would have me tearing my hair out for days, I'm sure; it's just not the kind of thing I'm good at. I know my weaknesses. :-> So it will wait until someone who has some experience with this stuff can spare the time.

[jeromekelleher]

Sorry @bhaller, I'm afraid I don't have time for this either right now. If you want to have a go at it, I'd use the bcftools recipe as a guide (using CMake for the building). As you have no dependencies it really should be straightforward. I'd be happy to help if you hit specific issues.

[jeromekelleher]

There's a PR open on conda-forge for a recipe here: conda-forge/staged-recipes#10613

conda-forge is the upstream for bioconda and has good support for Windows so I think it's a good place to put SLiM.

[bhaller]

There's a PR open on conda-forge for a recipe here: conda-forge/staged-recipes#10613

conda-forge is the upstream for bioconda and has good support for Windows so I think it's a good place to put SLiM.

What does it mean, in practice, that conda-forge is the upstream for bioconda? If SLiM is on conda-forge does that mean it is automatically available through bioconda also? Sorry, that's probably a dumb question, but I have never used conda or bioconda, and have only a very vague understanding of what they are.

[bhaller]

The initial recipe has been merged into conda-forge, so the slim-feedstock recipe should get created soon and the slim packages for linux/osx appear on anaconda after that.

OK, groovy. So if I understand correctly, this means that SLiM is now publicly available as an installable package through conda? I don't really know what the difference is between conda, conda-forge, anaconda and Bioconda; what would be a good/correct announcement to make publicly regarding this, and what link(s) should I supply in that announcement? Am I correct in thinking that being in conda-forge means it is in all of the others because conda-forge is "upstream" as you said? (Also, I think the link you supplied above for the anaconda page is identical to the link you gave for the "feedstock repo".)

Anyone fancy testing them out, perhaps doing a little perf comparison to see if this build is significantly different to a native one?

Sorry to be so useless on this, but I have no idea how I'd go about doing this without risking screwing up the configuration of my machine (leading to having to reinstall from backups, which is a nightmare I'd prefer to avoid). I normally use Macports to install stuff, and (as I understand it) it is not a good idea to mix and match different package installers. I imagine there's some way to sandbox testing like this, but I have no idea how. If somebody has a machine that is not their primary work machine, and would be trivial to wipe and reinstall if it gets screwed up, that would be better.

Hmm, there's somebody in the Messer Lab who I know uses conda and has some degree of expertise with it. I don't know his GitHub handle, though. I'll work on it.

Hmm, the package files are pretty corpulent at 2.5Mb. It'd be good to see what's in there and if it could be SLiMmed down (I'm here all week folks!).

If the "package files" include the sources, then 2.5Mb seems fairly slim, actually; even without the sources for SLiMgui, the SLiM source code alone is > 6 Mb. But if the sources are not part of it, then perhaps it is bloat, yes. :->

[bhaller]

I have just announced conda support on slim-announce and slim-discuss, so I think it makes sense to close this issue now. When the next version of SLiM is released, hopefully somebody can walk me through how to revise this to work with the new release, and then after that hopefully I can be self-sufficient. Thanks everybody!