GithubHelp home page GithubHelp logo

Comments (2)

fabiolab avatar fabiolab commented on July 28, 2024

As a clue.
A note from the elasticsearch python driver explains that using "fork()" may make things become that weird :

Is it at all possible that it is caused by a fork()? It might happen, if you have a connection open (which we do - we keep them open to be able to re use them) and you call fork() (or your container - uWSGI or celery - does) then both new processes share the same file descriptor and things might get weird.

Thus, Swallow multithreading system relies on the multiprocessing module, whom default mode is based on "fork()" :

     Depending on the platform, multiprocessing supports three ways to start a process. These start methods are

spawn

    The parent process starts a fresh python interpreter process. The child process will only inherit those resources necessary to run the process objects run() method. In particular, unnecessary file descriptors and handles from the parent process will not be inherited. Starting a process using this method is rather slow compared to using fork or forkserver.

    Available on Unix and Windows. The default on Windows.
fork

    The parent process uses os.fork() to fork the Python interpreter. The child process, when it begins, is effectively identical to the parent process. All resources of the parent are inherited by the child process. Note that safely forking a multithreaded process is problematic.

    Available on Unix only. The default on Unix.
forkserver

    When the program starts and selects the forkserver start method, a server process is started. From then on, whenever a new process is needed, the parent process connects to the server and requests that it fork a new process. The fork server process is single threaded so it is safe for it to use os.fork(). No unnecessary resources are inherited.

    Available on Unix platforms which support passing file descriptors over Unix pipes.

A solution may be to swap from fork to forkserver mode.

from swallow.

fabiolab avatar fabiolab commented on July 28, 2024

Done in 1.5.0

from swallow.

Related Issues (4)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.