I don't like run(shutdown=True) about curio

[dabeaz]

The kernel maintains a certain amount of system state behind the scenes. For example, a loopback socket connection, a thread pool, a process pool, some signal handling state, and so forth. The purpose of the shutdown argument is to control whether or not everything gets killed and resources released.

If you want to run a single coroutine and cleanup afterwards, you can use the much shorter curio.run(coro) function. In this case, you don't even directly create or manipulate a kernel.

There are cases where you might not want shutdown to occur though. For example, if you have synchronous code that's submitting a lot of coroutines to the kernel one after the other. For this, you could use a context manager::

with curio.Kernel() as kernel:
       kernel.run(task1())
       kernel.run(task2())
       ...

In this case, the kernel gets shutdown automatically upon completion of the context manager.

I'm not so keen on having a separate shutdown(). For one, the shutdown process requires the kernel to run in order to properly finalize coroutines. If the kernel is already running, it would be a strange thing to call from a coroutine since it would initiate some kind of weird recursive execution of the kernel. I also don't want to give the impression that a separate shutdown() method would be safe to call from separately executing threads or things like that (e.g., a separate thread suspending the execution of a curio kernel running in a different thread).

At the moment, I consider the shutdown=True flag to be a mostly hidden implementation detail. If you use the curio.run() function or the context manager interface, you'll just never use it.

[njsmith]

Ahhh okay that makes much more sense then. So to check I'm understanding correctly:

The docs say: "If shutdown is True, the kernel will cancel all daemonic tasks and perform a clean shutdown once all regular tasks have completed." In fact, the kernel will always cancel all daemonic tasks and perform a clean shutdown once all regular tasks have completed (if by "clean shutdown" we mean "make sure to clean up all coroutines", like I was interpreting it). If shutdown=True, then it will also clean up internal state like the thread pool.

In particular, we maintain the invariant that doing kernel.run(coro1); kernel.run(coro2) always acts the same as curio.run(coro1); curio.run(coro2) except that the former might be a bit more efficient because it can re-use resources like thread pools. But this resources are stateless, so there's no way for state to "leak" from one call to run to the next (modulo bugs or pathological code that like tries to mess around with the contents of sys inside a worker subprocess). Yes?

I agree that if that's how this works then the details of the shutdown API are not very important because almost no-one should touch them :-). (Except that the docs could make it clearer that most people don't need to care about this). But, wouldn't a more idomatic API for this:

• kernel.run(coro) -> runs a coroutine
• kernel.close() -> cleans up internal state, error to call while run is running, can't call run again afterwards
• kernel.__exit__ -> calls self.close()
• kernel.__del__ -> calls self.close() and maybe issues a ResourceWarning if not already closed, because people really should use with instead of relying on the garbage collector

?

[dabeaz]

kernel.run(coro1); kernel.run(coro2) is not the same as curio.run(coro1); curio.run(coro2). The difference concerns daemonic tasks. Any coroutine is allowed to spawn so-called daemonic tasks that run in the background (analogy: daemonic threads). If you do kernel.run(coro1) and coro1 terminates, any associated daemonic tasks remain active in the background. If you then perform kernel.run(coro2), those daemonic tasks will still be there ready to go. On the other hand, when you use curio.run(coro1) it performs a full shutdown that involves canceling all tasks (including daemonic tasks).

My rationale on daemonic tasks: One possible use of curio is as a kind of worker environment for servicing requests originating from synchronous code. A kernel could be created that has a complete execution environment set up including a thread pool, process pool, a collection of persistent daemonic worker tasks, and so forth. When you use kernel.run(coro), you're submitting a new task into the environment. Since the environment persists across kernel.run() methods, you don't need to constantly set it up or tear it down.

The top level curio.run(coro) is meant more for starting an entire application under curio. You give it the initial coroutine which might launch 100s of other tasks. The kernel runs until all of those tasks finish. When it's all done, the whole environment is torn down and the kernel is discarded.