conda-forge / distributed-feedstock

A conda-smithy repository for distributed.

About distributed-feedstock

Feedstock license: BSD-3-Clause

Home: https://distributed.dask.org

Package license: BSD-3-Clause

Summary: Distributed scheduler for Dask

Development: https://github.com/dask/distributed

Documentation: https://distributed.dask.org

Distributed is a lightweight library for distributed computing in Python. It extends both the concurrent.futures and dask APIs to moderate sized clusters.

Current build status

All platforms:

Current release info

Name	Downloads	Version	Platforms

Installing distributed

Installing distributed from the conda-forge channel can be achieved by adding conda-forge to your channels with:

conda config --add channels conda-forge
conda config --set channel_priority strict

Once the conda-forge channel has been enabled, distributed can be installed with conda:

conda install distributed

or with mamba:

mamba install distributed

It is possible to list all of the versions of distributed available on your platform with conda:

conda search distributed --channel conda-forge

or with mamba:

mamba search distributed --channel conda-forge

Alternatively, mamba repoquery may provide more information:

# Search all versions available on your platform:
mamba repoquery search distributed --channel conda-forge

# List packages depending on `distributed`:
mamba repoquery whoneeds distributed --channel conda-forge

# List dependencies of `distributed`:
mamba repoquery depends distributed --channel conda-forge

About conda-forge

conda-forge is a community-led conda channel of installable packages. In order to provide high-quality builds, the process has been automated into the conda-forge GitHub organization. The conda-forge organization contains one repository for each of the installable packages. Such a repository is known as a feedstock.

A feedstock is made up of a conda recipe (the instructions on what and how to build the package) and the necessary configurations for automatic building using freely available continuous integration services. Thanks to the awesome service provided by Azure, GitHub, CircleCI, AppVeyor, Drone, and TravisCI it is possible to build and upload installable packages to the conda-forge anaconda.org channel for Linux, Windows and OSX respectively.

To manage the continuous integration and simplify feedstock maintenance conda-smithy has been developed. Using the conda-forge.yml within this repository, it is possible to re-render all of this feedstock's supporting files (e.g. the CI configuration files) with conda smithy rerender.

For more information please check the conda-forge documentation.

Terminology

feedstock - the conda recipe (raw material), supporting scripts and CI configuration.

conda-smithy - the tool which helps orchestrate the feedstock. Its primary use is in the construction of the CI .yml files and simplify the management of many feedstocks.

conda-forge - the place where the feedstock and smithy live and work to produce the finished article (built conda distributions)

Updating distributed-feedstock

If you would like to improve the distributed recipe or build a new package version, please fork this repository and submit a PR. Upon submission, your changes will be run on the appropriate platforms to give the reviewer an opportunity to confirm that the changes result in a successful build. Once merged, the recipe will be re-built and uploaded automatically to the conda-forge channel, whereupon the built conda packages will be available for everybody to install and use from the conda-forge channel. Note that all branches in the conda-forge/distributed-feedstock are immediately built and any created packages are uploaded, so PRs should be based on branches in forks and branches in the main repository should only be used to build distinct package versions.

In order to produce a uniquely identifiable distribution:

• If the version of a package is not being increased, please add or increase the build/number.
• If the version of a package is being increased, please remember to return the build/number back to 0.

Feedstock Maintainers

• @charlesbluca
• @fjetter
• @jakirkham
• @jcrist
• @jrbourbeau
• @marcelotrevisani
• @martindurant
• @mrocklin
• @ogrisel
• @tomaugspurger

distributed-feedstock's Issues

@conda-forge-admin, please update version

Additional comment:

No response

Bump toolz/cytoolz version for next release

In dask/distributed#3579 we bumped our minimum version of toolz/cytoolz to 0.8.2. We should do the same here for the next release

Require pyyaml

It seems this is used for loading configuration files. So maybe it should be added as a requirement.

Bump minimum `zict` to 2.2.0

In dask/distributed#7709 we increased the minimum supported version of zict to 2.2.0. This is just a reminder to account for this here during the next release

@conda-forge-admin, please re-render

@conda-forge-admin, please re-render.

Note: Needed for conda-build 3 support.

Please, bump to 1.10.2 release

Latest version not availble on Linux64

On Linux64 machine:

conda search distributed -c conda-forge 

I get the following output(truncated):

                             1.12.0                   py34_0  defaults        
                          .  1.12.0                   py35_0  defaults        
                             1.12.1                   py27_0  conda-forge     
                             1.12.1                   py34_0  conda-forge     
                          .  1.12.1                   py35_0  conda-forge 

On OSX, i can see 1.12.2 however i cannot use it on Linux64 os.

Update dependencies in next release

This issue is a reminder to update the following dependencies for the next release:

• Minimum tblib version to 1.6.0 (xref dask/distributed#3451)
• For Python < 3.8 the minimum tornado version is 5, while for Python >= 3.8 we have a minimum tornado version of 6.0.3 (xref dask/distributed#3249)

We should add locket>=1.0 next release

Comment:

cc @jrbourbeau @jsignell

Add test for `dask spec`

In dask/distributed#7427 we added a new dask spec CLI. We should make sure to update the test commands here

distributed-feedstock/recipe/meta.yaml

Lines 56 to 63 in dca713c

	commands:
	- pip check
	- dask-scheduler --help
	- dask-ssh --help
	- dask-worker --help
	- dask scheduler --help
	- dask ssh --help
	- dask worker --help

in the next release

@conda-forge-admin, please add user @fjetter

Additional comment:

No response

Update `doc_source_url` to use `main`

Currently this URL specifies master, but we have renamed it to main. Would be good to update the URL here as well to reflect this

distributed-feedstock/recipe/meta.yaml

Line 76 in 823d9c5

doc_source_url: https://github.com/dask/distributed/blob/master/docs/source/index.rst

Add `packaging` dependency

In dask/distributed#5624 we added packaging as a requirement for distributed. We should be sure to add packaging >=20.0 during the next release here.

cc @jakirkham for visibility

Add setuptools

In dask/distributed#3305 we added an import of pkg_resources. So we should add setuptools (which includes pkg_resources) as a runtime dependency in the next release

Add jinja2 dependency

In dask/distributed#5188 we added jinja2 as a dependency to distributed. This is a reminder to add jinja2 here during the next release

Bump `click` dependency to be `>=7.0`

In dask/distributed#7226 we bumped out click version dependency to be click >=7.0. This is a reminder we should do the same thing here at the next release.

Bump `click` minimum to 8.0

Just a reminder for the next release. xref dask/distributed#7637

New version (1.18.1) depends on dask-core, breaks older dask versions

The 1.18.1 release depends directly on dask-core>=0.15.2.

The dask package was split into dask and dask-core at this version.

I pinned my version to dask=0.15.1 but when I then upgraded my conda environment, it installed distributed=1.18.2 which then installed dask-core.

Is this a bug? Should distributed only depend on dask, not dask-core directly?

Or is the solution to have dask-core incompatible with dask<=0.15.1?

mark build 0 of distributed 2.8.0 as broken

Issue:
I am getting dask 2.6.0 with distributed 2.8.0 via build 0 where the constraints on dask were not correct. I think build 0 needs to be moved to broken.

conda create -y -n test dask=2.6.0 distributed
conda activate test
python -c "import distributed.worker"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/xylar/miniconda3/envs/test/lib/python3.8/site-packages/distributed/__init__.py", line 3, in <module>
    from .actor import Actor, ActorFuture
  File "/home/xylar/miniconda3/envs/test/lib/python3.8/site-packages/distributed/actor.py", line 7, in <module>
    from .client import Future, default_client
  File "/home/xylar/miniconda3/envs/test/lib/python3.8/site-packages/distributed/client.py", line 68, in <module>
    from .worker import dumps_task, get_client, get_worker, secede
  File "/home/xylar/miniconda3/envs/test/lib/python3.8/site-packages/distributed/worker.py", line 21, in <module>
    from dask.system import CPU_COUNT
ModuleNotFoundError: No module named 'dask.system'

Environment (conda list):

$ conda list
# packages in environment at /home/xylar/miniconda3/envs/test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
bokeh                     1.4.0                    py38_0    conda-forge
ca-certificates           2019.9.11            hecc5488_0    conda-forge
certifi                   2019.9.11                py38_0    conda-forge
click                     7.0                        py_0    conda-forge
cloudpickle               1.2.2                      py_0    conda-forge
cytoolz                   0.10.1           py38h516909a_0    conda-forge
dask                      2.6.0                      py_0    conda-forge
dask-core                 2.6.0                      py_0    conda-forge
distributed               2.8.0                      py_0    conda-forge
freetype                  2.10.0               he983fc9_1    conda-forge
fsspec                    0.6.0                      py_0    conda-forge
heapdict                  1.0.1                      py_0    conda-forge
jinja2                    2.10.3                     py_0    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
libblas                   3.8.0               14_openblas    conda-forge
libcblas                  3.8.0               14_openblas    conda-forge
libffi                    3.2.1             he1b5a44_1006    conda-forge
libgcc-ng                 9.1.0                hdf63c60_0  
libgfortran-ng            7.3.0                hdf63c60_2    conda-forge
liblapack                 3.8.0               14_openblas    conda-forge
libopenblas               0.3.7                h6e990d7_3    conda-forge
libpng                    1.6.37               hed695b0_0    conda-forge
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                hc3755c2_1    conda-forge
locket                    0.2.0                      py_2    conda-forge
lz4-c                     1.8.3             he1b5a44_1001    conda-forge
markupsafe                1.1.1            py38h516909a_0    conda-forge
msgpack-python            0.6.2            py38hc9558a2_0    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.17.3           py38h95a1406_0    conda-forge
olefile                   0.46                       py_0    conda-forge
openssl                   1.1.1d               h516909a_0    conda-forge
packaging                 19.2                       py_0    conda-forge
pandas                    0.25.3           py38hb3f55d8_0    conda-forge
partd                     1.0.0                      py_0    conda-forge
pillow                    6.2.1            py38h6b7be26_0    conda-forge
pip                       19.3.1                   py38_0    conda-forge
psutil                    5.6.5            py38h516909a_0    conda-forge
pyparsing                 2.4.5                      py_0    conda-forge
python                    3.8.0                h357f687_3    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
pytz                      2019.3                     py_0    conda-forge
pyyaml                    5.1.2            py38h516909a_0    conda-forge
readline                  8.0                  hf8c457e_0    conda-forge
setuptools                41.6.0                   py38_1    conda-forge
six                       1.13.0                   py38_0    conda-forge
sortedcontainers          2.1.0                      py_0    conda-forge
sqlite                    3.30.1               hcee41ef_0    conda-forge
tblib                     1.4.0                      py_0    conda-forge
tk                        8.6.9             hed695b0_1003    conda-forge
toolz                     0.10.0                     py_0    conda-forge
tornado                   6.0.3            py38h516909a_0    conda-forge
wheel                     0.33.6                   py38_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zict                      1.0.0                      py_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge
zstd                      1.4.4                h3b9ef0a_1    conda-forge

Details about conda and system ( conda info ):

$ conda info

     active environment : test
    active env location : /home/xylar/miniconda3/envs/test
            shell level : 2
       user config file : /home/xylar/.condarc
 populated config files : /home/xylar/.condarc
          conda version : 4.7.12
    conda-build version : 3.18.11
         python version : 3.7.3.final.0
       virtual packages : 
       base environment : /home/xylar/miniconda3  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/xylar/miniconda3/pkgs
                          /home/xylar/.conda/pkgs
       envs directories : /home/xylar/miniconda3/envs
                          /home/xylar/.conda/envs
               platform : linux-64
             user-agent : conda/4.7.12 requests/2.22.0 CPython/3.7.3 Linux/4.15.0-1063-oem ubuntu/18.04.3 glibc/2.27
                UID:GID : 1001:1001
             netrc file : None
           offline mode : False

@conda-forge-admin, please re-render

@conda-forge-admin, please re-render.

To update the maintainer list in the README after this change ( #92 ).

Remove dask-submit and dask-remote in next release

We recently removed the dask-submit and dask-remote entrypoints in dask/distributed#3280. We should make sure to remove the following lines here for the next distributed release:

distributed-feedstock/recipe/meta.yaml

Lines 17 to 18 in 49195e1

	- dask-remote=distributed.cli.dask_remote:go
	- dask-submit=distributed.cli.dask_submit:go

This issue is just a reminder

automerge

@conda-forge-admin, please add bot automerge

cc @mrocklin

Update dependencies after `mindeps` CI build updates

In dask/distributed#7285 we updated some dependencies as part of adding anew mindeps CI build to distributed. We should make sure to align version constraints here to match during the next release.

cc @charlesbluca for visibility

@conda-forge-admin, please update version

Additional comment:

No response

Remove Click 8 constraint

In PR ( #165 ) we constrained click to be less than 8.0.0, but we can remove this in the next Distributed release as this issue has been fixed

distributed-feedstock/recipe/meta.yaml

Line 30 in 74c07c6

- click >=6.6,<8.0.0

cc @jrbourbeau (for vis)

Bump cloudpickle to 1.3.0

We recently bumped the cloudpickle requirement to 1.3.0. Would be good to bump it here as well.

cc @jrbourbeau (for awareness)

Remove recent push?

So apparently the recent release that we pushed out was missing some critical files. Is there a way to remove these packages from anaconda.org?

@conda-forge-admin, please re-render

@conda-forge-admin, please re-render.

Build with cython

I'm sure all the contributors here are across it, but just so the conda package doesn't get forgotten, it would be good for distributed to be as fast as possible and hence to be built with cython

xref: dask/distributed#4292

Disabling PyPy

Recently the Dask Community decided to drop PyPy support. Given this, what is the best way for us to disable PyPy support? We can add a skip to the recipe to disable these builds.

Though as discussed in issues ( regro/cf-scripts#1044 ) ( regro/cf-scripts#1386 ), we don't have a great way to opt-out of migrations and this can have downstream effects.

Given this, am raising this issue to solicit feedback ( cc @conda-forge/core ) on how we should handle this here and the proceed with next steps.

cc @jrbourbeau @mrocklin

Drop Python 3.7

In dask/distributed#5683 we dropped support for Python 3.7. This is a reminder to update our recipe accordingly for the next release. Additionally, @jakirkham mentioned dask/distributed#5683 (comment) we should also be able to switch back to noarch again

Remove openssl version constraint

This is a reminder to remove the openssl version constrain

distributed-feedstock/recipe/meta.yaml

Line 41 in 0c03d36

- openssl !=1.1.1e

for future releases once dask/distributed#3625 has been merged

Add contextvars for Python 3.6

Distributed now depends on contextvars when using Python 3.6. Would be good to add this dependency when updating to 2.16.0.

cc @jrbourbeau (for vis)

Pin openssl in next release

xref dask/distributed#3588

Add version ceiling for setuptools

Since setuptools is explicitly listed as a distributed requirement, it should have a ceiling: setuptools <60.0.0

ref: conda-forge/conda-forge-repodata-patches-feedstock#204

Require cytoolz?

Wondering if it makes sense to include cytoolz as a required run dependency given toolz is already required and cytoolz will help boost performance for some use cases.

Switch back to template friendly URLs

In PR ( #213 ), we had to switch to hashed URLs (see below) as the template friendly ones were downloading empty files. Please see upstream issue ( pypi/infra#104 ) for details. Would be good to revert back to the template friendly URLs as these are simpler for version updates and in working with other conda-forge tooling.

distributed-feedstock/recipe/meta.yaml

Line 9 in 2957d7f

url: https://files.pythonhosted.org/packages/57/f6/bb2532524de2bad548d5a0d731659389dc9fb3483b18715fa3c359c4111c/distributed-{{ version }}.tar.gz

Move openssl to run_constrained section in next release

For the next release we should move openssl to the run_constrained section (xref #122 (comment))