civisanalytics / datascience-python Goto Github PK

Hi there!

I attempted to build this docker image locally and the build failed with the following error:

Solving environment: ...working... failed

ResolvePackageNotFound:
  - openblas=0.3.5

The command '/bin/sh -c conda install -y boto &&     conda install -y nomkl &&     conda env update -f environment.yml -n root &&     conda clean --all -y &&     rm -rf ~/.cache/pip' returned a non-zero code: 1

This can be replicated by pulling down this repository and running docker build -t datascience-python .

When pip installing packages on top of the datascience-python docker image v4.0.1, the following warning shows up in the logging:

boto3 1.5.11 has requirement botocore<1.9.0,>=1.8.25, but you'll have botocore 1.5.38 which is incompatible.

As a sanity check, environment.yml indeed has boto3 pinned down at 1.5.11 and botocore at 1.5.38:

https://github.com/civisanalytics/datascience-python/blob/v4.0.1/environment.yml#L7-L9

Also, boto3 1.5.11 does specify botocore>=1.8.25,<1.9.0:

https://github.com/boto/boto3/blob/1.5.11/setup.py#L17

Instead of submitting a pull request right away, I thought I should create an issue first, just in case the incompatible dependencies between boto3 and botocore were intentional for reasons I'm unaware of, and also because there are likely other dependencies that we should probably update as well (it's been two months since the v4.0.1 release in early February 2018) but I don't know if the maintainers (@stephen-hoover?) are already working on it.

The latest release was 4.2.0 from ~8 months ago, which means lots of the dependency versions in environment.yml are not up-to-date by now. It's time to go through it with a fine-tooth comb and update the dependencies.

FWIW, I noticed this issue with cloudpickle, pinned at 0.5.2 through this datascience-python 4.2.0 Docker image, apparently unable to properly unpickle certain custom class objects; the problem was resolved after I force installed the latest cloudpickle 0.6.1.

@stephen-hoover for viz

There's probably no way to tell with certainty, but there's a good chance that some of the packages in environment.yml are seldom or never used in our current and foreseeable production environments, and therefore can be considered for removal to slim down the Docker image. Here's a list to start:

• beautifulsoup4
• dask
• nose
• urllib3 (requests is already included, though I don't know if production workflows depend on some crucial urllib3 features that requests doesn't support)
• dropbox
• python-simple-hipchat

Even if some of these packages are needed very occasionally, they can naturally be installed on the fly. In general, I feel like this ds-py image should only include packages that have at least medium (if not high) usage.

In the current README.md, we use relative links. This is nice, because as @stephen-hoover pointed out, the links are correct across branches and commits. However, since these links are relative and we use the same README.md for Dockerhub as for Github, the links work on Github but not on Dockerhub.