modin-project / unidist Goto Github PK

View Code? Open in Web Editor NEW

41.0 41.0 7.0 639 KB

Unified Distributed Execution

Home Page: https://unidist.readthedocs.io

License: Apache License 2.0

Python 99.14% Cython 0.31% C++ 0.54%

dask-distributed distributed mpi multiprocessing python ray

unidist's People

Contributors

Stargazers

Watchers

Forkers

devin-petersohn yarshev prutskov no-ponomarev retribution98 arunjose696 luweizheng

unidist's Issues

Align default value of REDIS_PASSWORD with Ray

The current default redis password (RayRedisPassword) is incompatible with the default value in Ray. It would be better to use default value from Ray:

from ray import ray_constants

ray_constants.REDIS_DEFAULT_PASSWORD

This makes possible to connect to the default Ray cluster without set `MODIN_REDIS_PASSWORD`

Put documentation in readthedocs

Add rst documentation for `unidist.config`

`mpi` backend failed on Census benchmark

Traceback (most recent call last):
  File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/worker.py", line 749, in <module>
    event_loop()
  File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/worker.py", line 675, in event_loop
    request = communication.recv_complex_operation(comm, source_rank)
  File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/communication.py", line 384, in recv_complex_operation
    return recv_complex_data(comm, source_rank)
  File "/localdisk/aprutsko/unidist/unidist/core/backends/mpi/core/communication.py", line 291, in recv_complex_data
    raw_buffers = [None] * raw_buffers_size
TypeError: can't multiply sequence by non-int of type 'list'

Add configs for `unidist` behavior tuning

We should provide some stuff like setting env vars for unidist behavior tuning.

Investigate hotspots for MPI backend on benchmarks

We should investigate hotspots for MPI backend on benchmarks (plasticc, cencus, taxi) and try fixing them.

A list of fixed performance issues/PRs:

Add mpi4py to pip distribution dependencies

Get rid of pandas dependency for MPI backend

Expand CLI options up to all config values

Add tests run to CI

This is a subtask of #9.

Use a single deps file both for developing and packaging

Current set is present as requirements.txt, it will be good to use deps from this file in conda env creating and setup.py without creating separate files to have one place with deps.

Add configs for Dask cluster setup

List of needed configs:

IsDaskCluster
DaskSchedulerAddress

Correct check for __reduce_ex__ atrribute

Use lower-case for `str` TypeDescriptor

Throw a user-friendly error message in case mpi4py is not installed

Add CLI to run `unidist` on various backends

Need to create a way to use all unidist backends by the same way. Also, this is needed for:

simplifying of running MPI backend
freeing a user from the need to know the internal approach of running unidist MPI scripts

Add black and flake8 to env files

`multiprocessing` backend incorrectly frees grabbed workers for actors in case a little amount of workers

Fix `test_general` hang in case `multiprocessing` backend is used

Tests in test_general.py hang in case of using a small number of workers.

unidist/unidist/test/test_general.py

Lines 15 to 24 in bd45958

 # FIXME: If we run all tests (i.e. pytest unidist/test), the tests are hanging. 

 # However, if we remove third argument, the tests will be passed. This is true for MultiProcessing backend and 

 # requires an investigation. 

 @pytest.mark.parametrize( 

  "object_ref", 

  [unidist.put(1), task.remote(3), TestActor.remote().task.remote(1)], 

  ids=["put", "task", "actor"], 

 ) 

 def test_is_object_ref(object_ref): 

 assert_equal(unidist.is_object_ref(object_ref), True)

Possible reason is in incorrect finishing worker-process in multiprocessing backend.

To reproduce need to set CpuCount = 2.

Add `ValueSource` enum to check where the `Parameter` value got from

Add `BackendName` class to `unidist/core/base/common.py` to use instead of direct string backend names

Set up CI for `unidist`

Add a contributing page for `unidist`

Should be similar to the contributing page in Modin - https://modin.readthedocs.io/en/latest/contributing.html.

Logo design

Update the repo with new logos

`What do these changes do` section in PRs description

Add documentation on CLI usage

[MPI] Multi-node performance benchmark investigation

Return `exit status` code via CLI

Rename UNIDIST_OBJECT_STORE_MEMORY to UNIDIST_RAY_OBJECT_STORE_MEMORY

Add dependencies for development and packaging

Update `High-Level Execution View` section in docs

The picture should be updated

scaleout -> unidist
img -> svg
python backend should be added to the picture

The link to MPIBackend should be updated.

Might not be limited by this.

Get rid of pandas dependency for Ray backend

Add parameters support in `unidist.init`

Parameters support should be added to unidist.init method to give more flexibility in setup framework.
Parameters are the next:

num_cpus
backend
address
...

C:\Users\aprutsko\repositories\unidist>python unidist\cli -m pytest unidist\test\test_general.py -b mpi
Windows fatal exception: access violation

Current thread 0x00005c44 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1174 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
  File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\communication.py", line 18 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\executor.py", line 13 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\core\__init__.py", line 5 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\aprutsko\repositories\unidist\unidist\core\backends\mpi\backend.py", line 10 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\aprutsko\repositories\unidist\unidist\core\base\utils.py", line 88 in get_backend_proxy
  File "C:\Users\aprutsko\repositories\unidist\unidist\api.py", line 34 in make_decorator
  File "C:\Users\aprutsko\repositories\unidist\unidist\api.py", line 118 in remote
  File "C:\Users\aprutsko\repositories\unidist\unidist\test\utils.py", line 40 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "C:\Users\aprutsko\repositories\unidist\unidist\test\test_general.py", line 10 in <module>
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\assertion\rewrite.py", line 170 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1014 in _gcd_import
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\importlib\__init__.py", line 127 in import_module
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\pathlib.py", line 524 in import_path
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 578 in _importtestmodule
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 500 in _getobj
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 291 in obj
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 516 in _inject_setup_module_fixture
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\python.py", line 503 in collect
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 341 in <lambda>
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 311 in from_call
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 341 in pytest_make_collect_report
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\runner.py", line 458 in collect_one_node
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 808 in genitems
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 634 in perform_collect
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 333 in pytest_collection
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 322 in _main
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 269 in wrap_session
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\config\__init__.py", line 162 in main
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\site-packages\pytest\__main__.py", line 5 in <module>
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\runpy.py", line 87 in _run_code
  File "C:\Users\aprutsko\.conda\envs\unidist\lib\runpy.py", line 194 in _run_module_as_main

===================================================================================

Add `unidist` to pip

Add high-level APIs to cover all required features in Modin

List of functions:

ray.cluster_resources()
ray.nodes()

The APIs may be either the same as above or similar and provide all required info.

All tests related to actors are failed for `multiprocessing` backend on Windows

Testing of actor for multiprocessing backend on Windows throwed the next errors:

    @pytest.mark.parametrize("is_use_options", [True, False])
    def test_address_space(is_use_options):
>       actor0 = TestActor.options().remote(0) if is_use_options else TestActor.remote(0)

unidist\test\test_actor.py:34:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
unidist\core\base\actor.py:189: in remote
    actor_cls = self._actor_cls(*self._actor_cls_args)._remote(*args, **kwargs)
unidist\core\backends\multiprocessing\actor.py:132: in _remote
    self._actor_handle = mp.Actor(self._cls, *args, **kwargs)
unidist\core\backends\multiprocessing\core\actor.py:107: in __init__
    manager.start()
..\..\.conda\envs\unidist\lib\multiprocessing\managers.py:579: in start
    self._process.start()
..\..\.conda\envs\unidist\lib\multiprocessing\process.py:121: in start
    self._popen = self._Popen(self)
..\..\.conda\envs\unidist\lib\multiprocessing\context.py:327: in _Popen
    return Popen(process_obj)
..\..\.conda\envs\unidist\lib\multiprocessing\popen_spawn_win32.py:93: in __init__
    reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <SpawnProcess name='BaseManager-15' parent=22516 initial>, file = <_io.BufferedWriter name=14>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       _pickle.PicklingError: Can't pickle <class 'unidist.test.utils.TestActor'>: it's not the same object as unidist.test.utils.TestActor

..\..\.conda\envs\unidist\lib\multiprocessing\reduction.py:60: PicklingError

	# FIXME: If we run all tests (i.e. pytest unidist/test), the tests are hanging.
	# However, if we remove third argument, the tests will be passed. This is true for MultiProcessing backend and
	# requires an investigation.
	@pytest.mark.parametrize(
	"object_ref",
	[unidist.put(1), task.remote(3), TestActor.remote().task.remote(1)],
	ids=["put", "task", "actor"],
	)
	def test_is_object_ref(object_ref):
	assert_equal(unidist.is_object_ref(object_ref), True)

modin-project / unidist Goto Github PK

unidist's People

Contributors

Stargazers

Watchers

Forkers

unidist's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs