Include mamba in miniforge? about miniforge

[scopatz]

I think you could also shove mamba into the base installer.

Also, what would be really nice is if you could just configure the condarc file to use mamba whenever conda is called

[dhirschfeld]

IMHO mamba is really coming along in terms of production-quality stability. I'm sure there are still a number of rough edges but the best way to handle that would be to get actionable feedback by making it (more) easily available by packaging it alongside conda in miniforge

[wolfv]

how about microforge with micromamba?

[nerohmot]

Who cares about this bit of disk space ?!?
What is more important is that things are installed easily and immediately as they should!
And face it, conda-forge is so huge that very quickly one starts craving for something faster ...

I also don't think it is a good idea to restrict mamba to be just a faster conda (cfr wolfv's comment above), no, mamba should have the freedom to offer other things in future too ! (I think here about things to do with Quetz, offering maybe a more efficient organizational architecture from which conda-forge itself could possibly also benefit from in future)

[rgommers]

I agree with the original rationale for this issue: mamba is way faster with conda-forge, and I consider conda effectively unusable for some use cases that require a lot of packages. Given that there's no significant improvements in conda on the horizon to make it approach mamba's performance, I'm very interested in (a) having mamba available by default and (b) maturing mamba faster through more usage. And mamba packages are only 500-600 kb in size. So I'd much prefer including it in baseline Miniforge.

[wolfv]

it would be possible to build a version of mamba that, like back in the days, uses conda for all downloading & extracting. Then we could drop all dependencies except for libsolv.

[wolfv]

conda install mamba -c conda-forge should work, and it's tested + used at least on Linux

[SylvainCorlay]

I opened mamba-org/mamba#81, mamba-org/mamba#82, mamba-org/mamba#83

We fixed:

• mamba-org/mamba#81
• mamba-org/mamba#82
• mamba-org/mamba#83

[rgommers]

If we are going to install mamba what other software should we include?

I think mamba is a special case (and I can't think of other equivalent special cases), for one because it replaces the most special package in Miniforge (conda), and for another because it should be in root, while we prefer to teach users to not fill root with other stuff.

[scopatz]

I think there is enough consensus to have a separate installer that we would feel comfortable merging in a PR to that effect.

I do think that there are two user stories going on here.

1. CI/CD where we want small and fast.
2. Downstream users who just want it to work, and don't care about speed or size. (Which is partially why the Anaconda distro exists)

We have 1 covered by miniforge and mambaforge. We don't really have 2 covered at all. I would be open to seeing proposals for a larger, more bloated, "just works" distribution. But that seems separate from the original issue here.

[isuruf]

It's always going to be a tradeoff between speed and storage. For binder, speed is important. For hosting in Heroku (where we host the webservice), storage is important. There's a 600MB limit in Heroku. (We switched to a different way in Heroku for the webservice and the storage is not an issue there anymore, but I've got another webapp using the older way)

I'd prefer if people don't put their needs above others and work on a solution for everybody.

[xhochy]

I made a Draft PR that would built Mambaforge in addition to Miniforge on this repo: #70

[rgommers]

+1 sounds interesting. Looks like you read someone's mind here @SylvainCorlay, we had a discussion on this exact topic a few days ago:)

To start with, are you thinking a set of installers based on mamba next to the current set based on conda? Could be nice to be able to evaluate them side-by-side.

[SylvainCorlay]

To start with, are you thinking a set of installers based on mamba next to the current set based on conda? Could be nice to be able to evaluate them side-by-side.

Just having it installed would allow people to use it, as an opt-in.

[isuruf]

mamba has a lot of different behaviour with conda that I don't want to replace. I used to use mamba and it updated my python version which took me a while to recover from and after 3 times I gave up.

[SylvainCorlay]

mamba has a lot of different behaviour with conda that I don't want to replace. I used to use mamba and it updated my python version which took me a while to recover from and after 3 times I gave up.

Do you have a minimal reproducer?

[SylvainCorlay]

Also, currently, from a fresh python 3.7 environment, conda install libsolv pybind11 pulls python 3.8, and conda install libsolv pybind11 python=3.7 does not give any conflict.

[isuruf]

Also, currently, from a fresh python 3.7 environment, conda install libsolv pybind11 pulls python 3.8, and conda install libsolv pybind11 python=3.7 does not give any conflict.

I guess you have a reproducer. conda doesn't upgrade your python unless explicitly told to do so.

[SylvainCorlay]

I guess you have a reproducer. conda doesn't upgrade your python unless explicitly told to do so.

this is conda. mamba behaves as expected on this one (and is much more reliable in my experience).

[isuruf]

I opened mamba-org/mamba#81, mamba-org/mamba#82, mamba-org/mamba#83

[isuruf]

And also mamba-org/mamba#38

[SylvainCorlay]

I opened mamba-org/mamba#81, mamba-org/mamba#82, mamba-org/mamba#83

thanks, I have responded to the individual tickets (marked 83 as a bug).

[rgommers]

mamba has a lot of different behaviour with conda that I don't want to replace

This is interesting (and being resolved), but it seems orthogonal to including mamba? You don't have to use it if conda is also still included.

[hmaarrfk]

What hardware specs did you have trouble with?

[hmaarrfk]

how do you currently install mamba? conda install mamba didn't work for me.

[hmaarrfk]

@SylvainCorlay i've been playing aroudn with mamba + miniforge for the last few weeks.

I noticed that it is recently poluting my directoris with pkgs folders. Any clue as to why that is happening?

[wolfv]

@hmaarrfk that was a bug which is fixed. What version of mamba are you using?

[hmaarrfk]

I guess I was on version 0.3.4. I guess I'm a typical "user" of "mamba", that never updates until they need to :D.

I can confirm that updating to 0.3.8 seems to alleviate the immiate issue (at least with a quick)

[nerohmot]

@wolfv as soon as TheSnakePit/mamba#83 is solved, i guess mamba can be added to miniforge right ?

[wolfv]

It would probably be possible to fix issue #83, although I think it's cleaner to install mamba in root.
Conda also does not work properly when not installed in the root environment.
And installing mamba in another environment will pull in a copy of conda (as mamba depends on conda).

[isuruf]

I'm still not convinced that we should include mamba in miniforge. miniforge is supposed to be the bare minimum needed to get conda and a python. Installing mamba is easy as conda install mamba. If we are going to install mamba what other software should we include?

[isuruf]

Perhaps build a miniforge-mamba?

[isuruf]

how about microforge with micromamba?

+1. I like the name micro because there'll be no python I suppose.

(a) having mamba available by default

replaces the most special package in Miniforge

Given that mamba is not a replacement, but an addition, people have to explicitly use mamba. For eg: they have to use mamba install numpy instead of conda install numpy. So, I don't know what's hard about doing conda install mamba.

for another because it should be in root, while we prefer to teach users to not fill root with other stuff.

Why should it be in root? conda can be in non-root environments.

[wolfv]

Why should it be in root? conda can be in non-root environments.

last time I tried, if you don't re-initialize the shell and try to create new environments with that conda, that will be trouble -- because conda checks it's own installation location and assumes that it is the root index.
Usually you'll never use the conda that's installed in an environment because conda is actually a bash function that will use the $CONDA_EXE that is configured in the bashrc, or something along those lines.

[isuruf]

And mamba packages are only 500-600 kb in size.

That's not true. You need these packages and would increase the size by 24 MB

    _openmp_mutex-4.5          |            0_gnu         435 KB  conda-forge
    icu-64.2                   |       h4c5d2ac_1        12.9 MB  conda-forge
    krb5-1.17.1                |       h09dc549_1         1.5 MB  conda-forge
    libarchive-3.3.3           |    hc47fbbf_1007         1.7 MB  conda-forge
    libcurl-7.71.1             |       hcdd3856_0         639 KB  conda-forge
    libedit-3.1.20191231       |       h46ee950_1         132 KB  conda-forge
    libgomp-7.5.0              |       h8e86211_6         702 KB  conda-forge
    libiconv-1.15              |    h6dd45c4_1006         2.0 MB  conda-forge
    libsolv-0.7.14             |       h8b12597_3         453 KB  conda-forge
    libssh2-1.9.0              |       hab1572f_3         227 KB  conda-forge
    libxml2-2.9.10             |       hb54214f_0         1.4 MB  conda-forge
    lz4-c-1.8.3                |    he1b5a44_1001         199 KB  conda-forge
    lzo-2.10                   |    h14c3975_1000         317 KB  conda-forge
    mamba-0.4.1                |   py37h6d3251c_0         628 KB  conda-forge
    zstd-1.4.4                 |       h3b9ef0a_2         895 KB  conda-forge

[isuruf]

Who cares about this bit of disk space ?!?

I do

[nerohmot]

@isuruf ... and what kind of systems are you working on then that you can't spare a hand full of MB's?!?
... a hand full of MB's that you will spend sooner or later anyway 😒

[dhirschfeld]

If package size is a deal-breaker for some applications then maybe a separate package could work - mambaforge (or similar). That way users could choose a leaner miniforge or a fatter mambaforge which also included mamba and it's deps by default.

[scopatz]

I think we absolutely should have a community distro that includes mamba or micromamba. If we don't want that in miniforge, having a mambaforge that is also built as part of this repo would be fine with me

[isuruf]

That's fine with me. I suggested it in #23 (comment), but looks like @rgommers and @nerohmot wants it included in miniforge.

[rgommers]

If we're doing Mambaforge that'd work for me too. I think it's quite a bit more work and less ideal though - letting users choose at download time rather than later on seems sub-optimal.

[isuruf]

I think it's quite a bit more work and less ideal though - letting users choose at download time rather than later on seems sub-optimal.

I don't get this. Users can choose to install mamba later on.

[rgommers]

Users can choose to install mamba later on.

I think you're ignoring the downsides of letting users jump through a hoop like that and letting them deal with two package management tools. It's like, why have an installer that defaults to conda-forge, when users can simply download Miniconda and change a config file to then default to conda-forge.

[nerohmot]

What are the chances of adding switches to the installer scripts ... something like :

• ./forge-install.sh --small ➜ smallest possible install (read: without mamba)
• ./forge-install.sh --medium ➜ small yet convenient base install (read: with mamba and maybe in future other stuff?)
• ./forge-install.sh --large ➜ big install where 95% of the users don't even need the available conda and mamba 😊 (read: the conda-forge alternative for anaconda)

[isuruf]

If somebody volunteers, I'm okay.

[nerohmot]

This ./forge-install.sh could be a 'front-end' for the current 8 (and growing i hope) individual packages there are now !
It could figure out the architecture (Linux/MacOS/... Intel/ARM/PPC ... 32/64 bit)on it's own, the install size from the switches, and the desired python implementation (CPython/ PyPy/...) also from switches ... based on the result download and install the exact installer.
@isuruf : do you also have size constraints on the size of the install script itself ?

[isuruf]

That's a different issue #39

[nerohmot]

two flies one stone 😉

[SylvainCorlay]

If we are going to install mamba what other software should we include?

I think mamba is a special case (and I can't think of other equivalent special cases), for one because it replaces the most special package in Miniforge (conda), and for another because it should be in root, while we prefer to teach users to not fill root with other stuff.

I think this is correct, which is that it goes alongside conda, and that we don't want to encourage users to install stuff in the base env.

[SylvainCorlay]

@minrk ran some benchmarks with mamba in repo2docker in a couple of repositories, and if we don't require a separate installation of mamba with conda, that will be a non-negligeable improvement in the binder build times (remove the green portion):

[SylvainCorlay]

I'd prefer if people don't put their needs above others and work on a solution for everybody

Really not what I was doing, but adding some material on the interest of the question.

It's always going to be a tradeoff between speed and storage.

Hopefully, a micromamba-style installer will help with both speed and size eventually.

[hmaarrfk]

There isn't too much automation in this repo.

Maybe Mamba can take on releasing minimamba3??

[hadim]

My two cents to the debate.

If disk size is an issue in certain cases I would expect speed to be really more important for the majority of the people (including me it's true but I don't think anybody here is trying to put their needs above others).

It's true you can always install miniforge and then conda install mamba but I think the main drawback of this is that you still rely on conda to solve conda install mamba and yes it is slow. On GA using setup-miniconda, the conda install mamba step takes 7min which corresponds to about 85% of my CI time. Shipping mamba within miniforge would probably reduce CI time for a lot of people (not only me here).

That being said those kinds of performance issues are also being looked at on the setup-miniconda side (see conda-incubator/setup-miniconda#106).

[hmaarrfk]

i'm becoming more and more convinced of mamba.

We still dont have mamba build, but maybe that shouldn't hold things up.

[hadim]

I am using it for a few months now. It has been a game changer for me.

[wolfv]

@hmaarrfk you can try boa, it has "conda mambabuild ..." and "boa build ..." where the boa native recipe format is slightly updated from the conda one. You can find working example recipes in the mamba-org/boa-forge repository if you're curious.

[hadim]

Thanks for the effort @SylvainCorlay and @xhochy. I can confirm CI with setup-conda is much faster now. See conda-incubator/setup-miniconda#89 (comment) for a snippet.

[SylvainCorlay]

We now have builds of miniforge available that include mamba (mambaforge).

Closing. Thanks everyone!