Methods passed accross multiprocessing needs to be importable by both processes about monkey

[ilija-lazoroski]

This is simplified multiissue.tar.zip. Unzip, untar and run python main.py. You should not have monkey-plugintoolbox installed anywhere.

[mssalvatore]

Solution

We decided on a different solution than the one described below. See #4126 (comment)

I was able to make the following changes to the main.py in mutliissue.tar and get it to work:

4a5
> from multiprocessing.context import BaseContext
47a49,56
> class CallableWrapper:
>     def __init__(self, func):
>         self._func = func
> 
>     def __call__(self):
>         return self._func()
> 
> 
49c58,59
<     def __init__(self, server: StubServer):
---
>     def __init__(self, context: BaseContext, server: StubServer):
>         self._context = context
53c63,65
<         self._server.register(transform)
---
>         SyncManager.register("CallableWrapper", CallableWrapper, exposed=["__call__"])
>         manager = self._context.Manager()
>         self._server.register(manager.CallableWrapper(transform))
66c78
<     stub_registrar = StubRegistrar(stub_server)
---
>     stub_registrar = StubRegistrar(stub_server, context)

Pros of this approach

• It works! At least, it does in this minimal example.
• The plugin remains ignorant of the underlying dependencies of multiprocessing.

Cons of this approach

• Every registration request results in a new SyncManager (process) being created

Analysis

While it's not ideal that new SyncManager processes are created per registration, it may be unavoidable. The manager process needs to be able to import the function, which means it needs to be started with the same import path as the plugin process. In order to keep plugins isolated, each plugin must therefore have its own manager process.

The new manager process appears to be automatically cleaned up when it is no longer needed.

In order to reduce the number of processes that get opened, we should modify the interface for IHTTPAgentBinaryServerRegistrar slightly. Currently, its reserve_download() method accepts an AgentBinaryTransform, which is a callable. By default, use_agent_binary() is passed. This function is essentially a NOP. Instead, the interface for reserve_download() should accept (and default to) None for the agent_binary_transform parameter. If agent_binary_transform is None, then no manager process needs to be started.

Tasks

• Implement the fix as shown above in the #Solution section of this comment. Validate that it solves the original problem.
• Validate that the sync manager processes get properly cleaned up when the plugin exits.
• Modify the IHTTPAgentBinaryServerRegistrar and any implementations in accordance with the ##Analysis section of this comment.

[mssalvatore]

The proposed solution didn't solve all of the issues. Rather than invest more time into this, we decided to solve this a different way. A callable that manipulates the binary is probably more heavy-handed than necessary. The majority of use cases can be covered by using a template instead of a callback.