GithubHelp home page GithubHelp logo

qxsch / workerpool Goto Github PK

View Code? Open in Web Editor NEW
101.0 101.0 21.0 5.12 MB

Parallel Processing WorkerPool for PHP

License: GNU General Public License v3.0

PHP 100.00%
parallel parallel-processing-workerpool php respawn semaphore worker-pool

workerpool's People

Contributors

cwhsu1984 avatar danielcosta avatar ejunker avatar furgas avatar kevenlibcast avatar qxsch avatar rubtsovav avatar zefrof avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

workerpool's Issues

Unreleased semaphores

We are using the WorkerPool for asynchronous tasks at work but we encounter a bug we cannot fix by ourselves.

After a while, we see this error message:

Warning: sem_get(): failed for key 0x24ffdec4: No space left on device in /home/[..]/vendor/qxsch/worker-pool/src/Semaphore.php on line 99

The ipcs -s command confirms that a bunch of semaphores are created, since several weeks, but not released.

It seems that semaphores created by the WorkerPool may not be released after that.
When does it occur, and how can we fix this?

Thank you for your work.

When using WorkerPool, PHP fails to flush output before exiting normally

I tried to use WorkerPool to parallelise some HTTP requests. It seems to work, except for one mysterious bug: when the main/parent PHP process exits, it fails to flush its buffer.

Example file:

<?php

require_once(__DIR__.'/vendor/autoload.php');

echo(htmlspecialchars(json_encode(['Starting...']))."<br>\n");
flush();

$pool = new \QXS\WorkerPool\WorkerPool();
$pool->setWorkerPoolSize(1);
$pool->disableSemaphore();
$pool->create(new \QXS\WorkerPool\ClosureWorker(function (int $value, \QXS\WorkerPool\Semaphore $semaphore, \ArrayObject $storage) : int {
    return $value * 2;
}));

foreach ([1, 2, 3] as $input) {
    $pool->run($input);
}

$pool->waitForAllWorkers();

foreach ($pool as $return) {
    echo(htmlspecialchars(json_encode([$return]))."<br>\n");
    flush();
}

echo(htmlspecialchars(json_encode(['Finished!']))."<br>\n");
flush();

echo(htmlspecialchars(json_encode(['Post-flush content.']))."<br>\n");

When visiting the URL of the test file, it outputs the following:

["Starting..."]
[{"pid":30,"data":2}]
[{"pid":30,"data":4}]
[{"pid":30,"data":6}]
["Finished!"]

Note that the content echoed after the last flush() is not present in the output. It seems the PHP process has exited without calling a final flush().

It's being run with PHP Apache. For the record, this is the Dockerfile being used (it has all the required extensions): https://github.com/dxw/wpc/blob/309265e25b4eccdbcf8350d0cf77d5c3c2e4ecba/images/wordpress/Dockerfile

Is there anything that can be done to fix this, other than by calling flush() after all content has been output?

Thanks!

All workers were gone. message in PHP 7.3

It crashes in the getNextFreeWorker method of the Workerpool.php file. I tried to debug a bit, but not sure if this has to do with PHP 7.3 specific. I upgraded PHP form 7.2 to 7.3 with Homebrew on my MacBookPro yesterday and after that it started. Weirdly enough on Ubuntu I already have it running for some time and didn't seem to have that.

Will be support PHP 8.3?

found some error when run on php 8.3. I test with simple https://github.com/qxsch/WorkerPool#a-more-sophisticated-example
doesn't work.

PHP Fatal error:  Uncaught Error: Call to undefined function QXS\WorkerPool\sem_get() in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php:99
Stack trace:
#0 /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/WorkerPool.php(268): QXS\WorkerPool\Semaphore->create()
#1 /mnt/d/dev/Patrickz/test/test.php(59): QXS\WorkerPool\WorkerPool->create()
#2 {main}
  thrown in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php on line 99
PHP Fatal error:  Uncaught QXS\WorkerPool\SemaphoreException: Semaphore hasn't yet been created. in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php:186
Stack trace:
#0 /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/WorkerPool.php(443): QXS\WorkerPool\Semaphore->destroy()
#1 /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/WorkerPool.php(390): QXS\WorkerPool\WorkerPool->destroy()
#2 [internal function]: QXS\WorkerPool\WorkerPool->onShutDown()
#3 {main}
  thrown in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php on line 186

PHP 8.3.0RC1 (cli) (built: Sep 2 2023 06:34:23) (NTS)

catch workerException

Will be good if we can catch of the worker exceptions. For example:

$wp = new WorkerPool();
$wp->onWorkerException(function(\Exception $ex, Worker $worker) {
   // there we can handle the exception (logging, restart the task or other)
});

Documentation: system requirements.

Glancing over some of the documentation it's not clear to me what the exact requirements are.
Would be nice to include those in the readme/wiki.

invalid Socket resources

Hi!

When all jobs are completed and the script is about the complete I get several warnings about invalid socket resources.

Warning: socket_last_error(): 420 is not a valid Socket resource /home/[...]/vendor/qxsch/worker-pool/src/SimpleSocket.php 159
Warning: socket_last_error(): 420 is not a valid Socket resource /home/[...]/vendor/qxsch/worker-pool/src/SimpleSocket.php 161

All seems to work properly nonetheless. I run Ubuntu 15 with PHP 5.6.4.
Any ideas on what the origin of those warnings could be?

Many thanks in advance!

Floris

No response when testing with PHPUnit

I am using qxsch/WorkerPool.
Thanks to you it is very useful. Thank you.

I am using PHPUnit to test where WorkerPool is used.
When the test is run, the results are not displayed.
Perhaps the execution has stopped and is no longer responding.

% ./vendor/bin/sail exec laravel.test php artisan test

   PASS  Tests\Unit\ExampleTest
  โœ“ that true is true
# (No longer responding)

# (in docker container)
% ps -aux 
root        36  0.8  2.6 142896 52908 pts/0    S+   12:33   0:00 phpunit: Parent
root        37  0.0  1.2 142896 26296 pts/0    S+   12:33   0:00 phpunit: Worker 1 of QXS\WorkerPool\ClosureWorker [free]
# (No change since then)

A Repository of projects that have been verified to work is temporarily open to the public.
Am I missing something?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.