saghul / python-fibers Goto Github PK

Create a pure Python implementation of 'fibers', on top of _continuation.continulet.

Current documentation for Fiber.throw() says:

Suspend the current running fiber and switch execution to the target fiber, raising the specified exception immediately. The fiber which called switch on this fiber earlier will now get an exception.

I believe that the second part is incorrect. The exception is propagated to the parent of the target fiber, instead of the fiber that switched into it. These may or may not be the same.

I'm interested in lightweight forking of fibers, with the intent of implementing reentrant coroutines, aka continuations, aka call/cc. Best attempt so far:

import copy
import fibers

def test_fork():
    def fn1():
        print("fn1")
        x = f2.switch(fibers.current())
        print("fn1", x)
        f2.switch(x)

    def fn2():
        crt = f1.switch()
        # TypeError: cannot serialize 'fibers._cfibers.Fiber' object
        # crt2 = copy.deepcopy(crt)
        a = crt.switch('a')
        print('result' , a)
        # b = crt2.switch('b')
        # print('result', b)

    f1 = fibers.Fiber(fn1)
    f2 = fibers.Fiber(fn2)
    f2.switch()

Is this doable?

Hey there,

thanks for this package! I've recently started working on a codebase that uses fibers , the project uses Python 3.10.

When installing pip is unable to build the wheel, with the following error:

Building wheels for collected packages: fibers
  Building wheel for fibers (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for fibers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [52 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/fibers
      copying fibers/__init__.py -> build/lib.linux-x86_64-cpython-310/fibers
      copying fibers/_pyfibers.py -> build/lib.linux-x86_64-cpython-310/fibers
      running build_ext
      building 'fibers._cfibers' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/src
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/venv310/include -I/py310/Python-3.10.14/Include -I/py310/Python-3.10.14 -c src/fibers.c -o build/temp.linux-x86_64-cpython-310/src/fibers.o
      src/fibers.c: In function ‘stacklet__callback’:
      src/fibers.c:225:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
        225 |     tstate->exc_type = NULL;
            |             ^~~~~~~~
            |             curexc_type
      src/fibers.c:226:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
        226 |     tstate->exc_value = NULL;
            |             ^~~~~~~~~
            |             curexc_value
      src/fibers.c:227:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
        227 |     tstate->exc_traceback = NULL;
            |             ^~~~~~~~~~~~~
            |             curexc_traceback
      src/fibers.c: In function ‘do_switch’:
      src/fibers.c:287:36: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
        287 |     current->ts.exc_type = tstate->exc_type;
            |                                    ^~~~~~~~
            |                                    curexc_type
      src/fibers.c:288:37: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
        288 |     current->ts.exc_value = tstate->exc_value;
            |                                     ^~~~~~~~~
            |                                     curexc_value
      src/fibers.c:289:41: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
        289 |     current->ts.exc_traceback = tstate->exc_traceback;
            |                                         ^~~~~~~~~~~~~
            |                                         curexc_traceback
      src/fibers.c:329:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
        329 |     tstate->exc_type = current->ts.exc_type;
            |             ^~~~~~~~
            |             curexc_type
      src/fibers.c:330:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_value’; did you mean ‘curexc_value’?
        330 |     tstate->exc_value = current->ts.exc_value;
            |             ^~~~~~~~~
            |             curexc_value
      src/fibers.c:331:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
        331 |     tstate->exc_traceback = current->ts.exc_traceback;
            |             ^~~~~~~~~~~~~
            |             curexc_traceback
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for fibers
Failed to build fibers
ERROR: Could not build wheels for fibers, which is required to install pyproject.toml-based projects

Neither the project I am working on, nor fibers have a pyproject.toml so I'm slightly confused by the last line on the log.

I tried --no-cache-dir --no-binary==fibers but still doesn't work, any advice?

I'm having trouble with multi-threading and fibers. I don't really have a standalone reproducer yet at the moment, but I'm hoping you can help me troubleshoot a bit.

What I'm doing is to run two concurrent threads via pyuv.Loop.queue_work, in addition to my main thread. Each of the additional threads has two fibers: one fibers runs an event loop, and the other does some potentially blocking or CPU intensive work.

I've done a bit printf() debugging, and what I've found out so far is the following: I'm experiencing that when one thread exits, the other thread will create a new main fiber in update_current(). This will allocate a new Fiber object with a new thread_h attribute. This prevents me from switching to a different fiber that is running in the same OS thread (confirmed via the gettid() system call), but according to Fiber_func_switch() is not running in the same OS thread.

So far I haven't been able to determine yet why the new main fiber is created. The Python side seems to be OK, because PyThreadState_GetDict returns the same dictionary for which previously a main fiber was created already.

Do you have any ideas or suggestions? Also I'm not completely following the logic how _global_state works, especially with regards to threading. Maybe you could elaborate on that a bit?

It appears that #7 does not fully fix the switching issues that I see. It's not a bug in the implementation, but a behaviour that in my view is not optimal in combination with external threads, like those offered by the libuv thread pool.

In Gruvi, a user can use the libuv threadpool (via pyuv) to run work that may block or that is CPU intenstive. In Gruvi, every thread has a special fiber that is called the Hub and which acts as the per-thread fiber scheduler. The work function can sometimes cause a Hub to be created, e.g. when a Gruvi API call is done.

I store the hub as a property on threading.current_thread(). The idea is that it's this prevents me from creating and destroying a Hub (including its event loop) every time a work function is run, which would be very expensive.

When the work function exits, the thread state dictionary is destroyed by threadpool_work_cb that calls PyGILState_Release. Because python-fibers uses the thread state dict to store the current fiber, this destroy the reference to the current fiber. Next time a work function is run on the same thread, a new main fiber will be created, with a new thread_h. This makes it impossible to switch from this new fiber, to the hub that was created on the same thread, but with a different thread_h.

One possible fix would be to use the alternative approach to store the current fiber using the Python TLS API. I think I would prefer that one. A slight drawback here is that it will be a bit slower on Python 2.7 that doesn't have native TLS. On Python 3.3 it will be as fast or a bit faster even.

Note that threading.local uses the thread dict as well. This means that attributes set on a local object will be not available from between two work functions running on the same thread in the thread pool, unlike attributes on threading.current_thread, which are available!

@saghul, I'm looking into the 3.11, 3.12 support and it's becoming obvious that the cpython engine changes every few releases in a way that breaks fibers.c/h (3.7 needed changes, 3.11 will need changes, and 3.12 will need still more changes)

I read your why.rst doc and sympathize with the reasons you initially created fibers. And I noticed that it compares itself to greenlet's behaviors (a package, incidentally, that I was unaware of when I initially went searching for a fibers package). It does nearly the same thing but with some specific set behaviors.

But I'm wondering if it might be worthwhile to make fibers into simply a pure python wrapper around the greenlet package? I haven't explored it fully yet, but I would be willing to explore how all the issues could be worked around, and keep the same behavior of your package, verified via the unit tests.

My only question is about performance: Will it perform horribly worse, or will it be quite acceptable? We could add a benchmark unit test to compare the two approaches.

Also you mention that you went with stacklet because it performs the save and restore operations within the asm which is better because of compiler behaviors. But I'd think it okay to just let the greenlet maintainers worry about those issues since it's an active and well-maintained library (w/ new release only 5 days ago).

If possible, even if there's just a slight performance penalty, wouldn't it be worth it to avoid the hassle of always chasing inevitable changes to the internals of cpython?

Thoughts?

In pypy you have a low level access to stacklet allowing you to pickle the frame and such things. would be interresting to have it as wll in fibers :)

In python 3.7, there has been some refactoring around exception tracking in pystate.h for coroutinues.

PR-21 goes with this ticket.

Currently fibers only ships with the necessary object file for linking in 32bit Windows systems. I need to include the one for 64bit Windows, but I don't have the setup to build it.

If you have a 64bit Windows 7 installation and are willing to help (it should be a simple task) ping me! :-)