saghul / python-fibers

Lightweight cooperative microthreads for Python

fibers: lightweight concurrent multitasking

Overview

Fibers are lightweight primitives for cooperative multitasking in Python. They provide means for running pieces of code that can be paused and resumed. Unlike threads, which are preemptively scheduled, fibers are scheduled cooperatively, that is, only one fiber will be running at a given point in time, and no other fiber will run until the user explicitly decides so.

When a fiber is created it will not run automatically. A fiber must be 'switched' into for it to run. Fibers can switch control to other fibers by way of the switch or throw functions, which switch control or raise and exception in the target fiber respectively.

Example:

import fibers

def func1():
    print "1"
    f2.switch()
    print "3"
    f2.switch()

def func2():
    print "2"
    f1.switch()
    print "4"

f1 = fibers.Fiber(target=func1)
f2 = fibers.Fiber(target=func2)
f1.switch()

The above example will print "1 2 3 4", but the result was obtained by the cooperative work of 2 fibers yielding control to each other.

CI status

Documentation

http://readthedocs.org/docs/python-fibers/

Installing

fibers can be installed via pip as follows:

pip install fibers

Building

Get the source:

git clone https://github.com/saghul/python-fibers

Linux:

./build_inplace

Mac OSX:

(XCode needs to be installed)
export ARCHFLAGS="-arch x86_64"
./build_inplace

Microsoft Windows:

python setup.py build_ext --inplace

Running the test suite

The test suite can be run using pytest:

python -m pytest -v .

Author

Saúl Ibarra Corretgé <[email protected]>

This project would not have been possible without the previous work done in the greenlet and stacklet (part of PyPy) projects.

License

Unless stated otherwise on-file fibers uses the MIT license, check LICENSE file.

Supported Python versions

Python >= 3.7 are supported. Other older Python versions might work, but they are not actively tested. CPython and PyPy are supported.

Supported architectures

x86, x86-64, ARM, MIPS64, PPC64 and s390x are supported.

Contributing

If you'd like to contribute, fork the project, make a patch and send a pull request. Have a look at the surrounding code and please, make yours look alike.

python-fibers's Issues

Forking fibers?

I'm interested in lightweight forking of fibers, with the intent of implementing reentrant coroutines, aka continuations, aka call/cc. Best attempt so far:

import copy
import fibers

def test_fork():
    def fn1():
        print("fn1")
        x = f2.switch(fibers.current())
        print("fn1", x)
        f2.switch(x)

    def fn2():
        crt = f1.switch()
        # TypeError: cannot serialize 'fibers._cfibers.Fiber' object
        # crt2 = copy.deepcopy(crt)
        a = crt.switch('a')
        print('result' , a)
        # b = crt2.switch('b')
        # print('result', b)

    f1 = fibers.Fiber(fn1)
    f2 = fibers.Fiber(fn2)
    f2.switch()

Is this doable?

thow(): documentation issue

Current documentation for Fiber.throw() says:

Suspend the current running fiber and switch execution to the target fiber, raising the specified exception immediately. The fiber which called switch on this fiber earlier will now get an exception.

I believe that the second part is incorrect. The exception is propagated to the parent of the target fiber, instead of the fiber that switched into it. These may or may not be the same.

Does not compile against python 3.7

In python 3.7, there has been some refactoring around exception tracking in pystate.h for coroutinues.

PR-21 goes with this ticket.

Fibers and the libuv thread pool

It appears that #7 does not fully fix the switching issues that I see. It's not a bug in the implementation, but a behaviour that in my view is not optimal in combination with external threads, like those offered by the libuv thread pool.

In Gruvi, a user can use the libuv threadpool (via pyuv) to run work that may block or that is CPU intenstive. In Gruvi, every thread has a special fiber that is called the Hub and which acts as the per-thread fiber scheduler. The work function can sometimes cause a Hub to be created, e.g. when a Gruvi API call is done.

I store the hub as a property on threading.current_thread(). The idea is that it's this prevents me from creating and destroying a Hub (including its event loop) every time a work function is run, which would be very expensive.

When the work function exits, the thread state dictionary is destroyed by threadpool_work_cb that calls PyGILState_Release. Because python-fibers uses the thread state dict to store the current fiber, this destroy the reference to the current fiber. Next time a work function is run on the same thread, a new main fiber will be created, with a new thread_h. This makes it impossible to switch from this new fiber, to the hub that was created on the same thread, but with a different thread_h.

One possible fix would be to use the alternative approach to store the current fiber using the Python TLS API. I think I would prefer that one. A slight drawback here is that it will be a bit slower on Python 2.7 that doesn't have native TLS. On Python 3.3 it will be as fast or a bit faster even.

Note that threading.local uses the thread dict as well. This means that attributes set on a local object will be not available from between two work functions running on the same thread in the thread pool, unlike attributes on threading.current_thread, which are available!

new approach

@saghul, I'm looking into the 3.11, 3.12 support and it's becoming obvious that the cpython engine changes every few releases in a way that breaks fibers.c/h (3.7 needed changes, 3.11 will need changes, and 3.12 will need still more changes)

I read your why.rst doc and sympathize with the reasons you initially created fibers. And I noticed that it compares itself to greenlet's behaviors (a package, incidentally, that I was unaware of when I initially went searching for a fibers package). It does nearly the same thing but with some specific set behaviors.

But I'm wondering if it might be worthwhile to make fibers into simply a pure python wrapper around the greenlet package? I haven't explored it fully yet, but I would be willing to explore how all the issues could be worked around, and keep the same behavior of your package, verified via the unit tests.

My only question is about performance: Will it perform horribly worse, or will it be quite acceptable? We could add a benchmark unit test to compare the two approaches.

Also you mention that you went with stacklet because it performs the save and restore operations within the asm which is better because of compiler behaviors. But I'd think it okay to just let the greenlet maintainers worry about those issues since it's an active and well-maintained library (w/ new release only 5 days ago).

If possible, even if there's just a slight performance penalty, wouldn't it be worth it to avoid the hassle of always chasing inevitable changes to the internals of cpython?

Thoughts?

fibers._cfibers.error: cannot switch to a Fiber on a different thread

I'm having trouble with multi-threading and fibers. I don't really have a standalone reproducer yet at the moment, but I'm hoping you can help me troubleshoot a bit.

What I'm doing is to run two concurrent threads via pyuv.Loop.queue_work, in addition to my main thread. Each of the additional threads has two fibers: one fibers runs an event loop, and the other does some potentially blocking or CPU intensive work.

I've done a bit printf() debugging, and what I've found out so far is the following: I'm experiencing that when one thread exits, the other thread will create a new main fiber in update_current(). This will allocate a new Fiber object with a new thread_h attribute. This prevents me from switching to a different fiber that is running in the same OS thread (confirmed via the gettid() system call), but according to Fiber_func_switch() is not running in the same OS thread.

So far I haven't been able to determine yet why the new main fiber is created. The Python side seems to be OK, because PyThreadState_GetDict returns the same dictionary for which previously a main fiber was created already.

Do you have any ideas or suggestions? Also I'm not completely following the logic how _global_state works, especially with regards to threading. Maybe you could elaborate on that a bit?

64bit Windows support

Currently fibers only ships with the necessary object file for linking in 32bit Windows systems. I need to include the one for 64bit Windows, but I don't have the setup to build it.

If you have a 64bit Windows 7 installation and are willing to help (it should be a simple task) ping me! :-)

low level access to stacklet

In pypy you have a low level access to stacklet allowing you to pickle the frame and such things. would be interresting to have it as wll in fibers :)

PyPy support

Create a pure Python implementation of 'fibers', on top of _continuation.continulet.