Resulting slug exceeds 300M limit about herokucondascipyflaskapp

[mmadsen]

Oh sure, here is it. All I did was add the line "nomkl" to your conda-requirements.txt. I wanted to make the minimal change possible.

nomkl
python=3.5.1
flask=0.10.1
itsdangerous=0.24
jinja2=2.8
markupsafe=0.23
numpy=1.10.1
pip=8.0.1
scipy=0.16.0
scikit-learn=0.17
setuptools=19.4
werkzeug=0.11.3
wheel=0.26.0

Thanks for talking with me about it. No worries on the confusion between msvc and mkl. noticing the MKL package size stimulated some thinking -- the difficulty in using optimized libraries here makes it difficult to do serious machine learning workers on a heroku worker in python, but it's very possible that such work is best done outside the fixed dyno structure of heroku anyway. But I'm still working on a worker app that will have just the numpy/scipy/sklearn stuff, and perhaps talks to a web app via listening to a task broker, such as Celery/RabbitMQ etc.

appreciate your help!

[arose13]

I'll check it again for you. The biggest problem was including a dependency that only runs on Microsoft ,'msvc' or something like that, so just make sure you don't have that included. Other than that I will redeploy this on Heroku and see if the problem has reborn itself.

[arose13]

@mmadsen I cannot seem to replicate the problem. I'm getting a slug size of 159MB. Are you also using this buildpackURL https://github.com/kennethreitz/conda-buildpack.git ?

[mmadsen]

Yes:

mark:HerokuCondaScipyFlaskApp/ (master) $ heroku buildpacks [21:21:11]
=== mmadsen-conda-test2 Buildpack URL
https://github.com/kennethreitz/conda-buildpack.git

The conda-requirements.txt:

python=3.5.1
flask=0.10.1
jinja2=2.8
numpy=1.10.1
pip=8.1.1
scipy=0.16.0
scikit-learn=0.17
setuptools=19.4
werkzeug=0.11.3
wheel=0.26.0

What I get in pushing:

remote: -----> Compressing...
remote: ! Compiled slug size: 458.1M is too large (max is 300M).
remote: ! See: http://devcenter.heroku.com/articles/slug-size

I've used the repo add on to clean the cache, and I'm stumped. Doesn't seem like it ought to be that large....

[mmadsen]

Out of curiosity to see if it was my local environment, I moved to a clean Linux server running CentOS 7, where I had never done any Heroku work, installed the toolbelt, cloned your repository, created an app using the buildpack, didn't touch any of the files, and did a push and got 309.7M. Am I missing a step?

[mmadsen]

Quick question. When I look at the details of the build packages, I'm consistently getting the MKL libraries package, which accounts for almost 120MB of that total slug size. When you build, and get a total of 159MB, does it include the MKL? I don't see how it could.

It would be nice to have MKL of course, because the whole point of having a numpy/scipy/conda worker is to do fast math, but it may not be possible in 300MB. But I'm wondering why mine would have MKL and other builds not, if that's what's going on here....thanks for looking at this.

[mmadsen]

SOLVED! It's definitely the MKL library, which is really, really too bad. If I add the directive "nomkl" to the top of the conda-requirements.txt, it causes the conda installer to only load non-MKL versions of numpy and scipy, and leave the MKL libraries and packages out of the build. I get a build size of 85M with a clean cache.

So it's definitely possible to build an Anaconda python web process or worker, but at the moment you can't build one that uses numpy/scipy linked to the Intel Math Kernel Libraries. Which limits performance if you wanted a worker doing machine learning algorithms, but oh well. I'll keep working on it - maybe I can get a minimal worker-only app with just miniconda, numpy, scipy, sklearn compiled and keep it under 300M -- only need to drop another 10-15MB.

Thanks!

[arose13]

That did remind me that both MSVC and MKL were both problems that I was encountering at different times on different projects. I am still surprised I wasn't able to replicate this error. Also could you post what your conda-requirements.txt looks like now please? Or you can make a pull request and I'll merge it.