qxsch / workerpool Goto Github PK
View Code? Open in Web Editor NEWParallel Processing WorkerPool for PHP
License: GNU General Public License v3.0
Parallel Processing WorkerPool for PHP
License: GNU General Public License v3.0
We are using the WorkerPool for asynchronous tasks at work but we encounter a bug we cannot fix by ourselves.
After a while, we see this error message:
Warning: sem_get(): failed for key 0x24ffdec4: No space left on device in /home/[..]/vendor/qxsch/worker-pool/src/Semaphore.php on line 99
The ipcs -s
command confirms that a bunch of semaphores are created, since several weeks, but not released.
It seems that semaphores created by the WorkerPool may not be released after that.
When does it occur, and how can we fix this?
Thank you for your work.
I tried to use WorkerPool to parallelise some HTTP requests. It seems to work, except for one mysterious bug: when the main/parent PHP process exits, it fails to flush its buffer.
Example file:
<?php
require_once(__DIR__.'/vendor/autoload.php');
echo(htmlspecialchars(json_encode(['Starting...']))."<br>\n");
flush();
$pool = new \QXS\WorkerPool\WorkerPool();
$pool->setWorkerPoolSize(1);
$pool->disableSemaphore();
$pool->create(new \QXS\WorkerPool\ClosureWorker(function (int $value, \QXS\WorkerPool\Semaphore $semaphore, \ArrayObject $storage) : int {
return $value * 2;
}));
foreach ([1, 2, 3] as $input) {
$pool->run($input);
}
$pool->waitForAllWorkers();
foreach ($pool as $return) {
echo(htmlspecialchars(json_encode([$return]))."<br>\n");
flush();
}
echo(htmlspecialchars(json_encode(['Finished!']))."<br>\n");
flush();
echo(htmlspecialchars(json_encode(['Post-flush content.']))."<br>\n");
When visiting the URL of the test file, it outputs the following:
["Starting..."]
[{"pid":30,"data":2}]
[{"pid":30,"data":4}]
[{"pid":30,"data":6}]
["Finished!"]
Note that the content echoed after the last flush()
is not present in the output. It seems the PHP process has exited without calling a final flush()
.
It's being run with PHP Apache. For the record, this is the Dockerfile being used (it has all the required extensions): https://github.com/dxw/wpc/blob/309265e25b4eccdbcf8350d0cf77d5c3c2e4ecba/images/wordpress/Dockerfile
Is there anything that can be done to fix this, other than by calling flush()
after all content has been output?
Thanks!
It crashes in the getNextFreeWorker
method of the Workerpool.php
file. I tried to debug a bit, but not sure if this has to do with PHP 7.3 specific. I upgraded PHP form 7.2 to 7.3 with Homebrew on my MacBookPro yesterday and after that it started. Weirdly enough on Ubuntu I already have it running for some time and didn't seem to have that.
found some error when run on php 8.3. I test with simple https://github.com/qxsch/WorkerPool#a-more-sophisticated-example
doesn't work.
PHP Fatal error: Uncaught Error: Call to undefined function QXS\WorkerPool\sem_get() in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php:99
Stack trace:
#0 /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/WorkerPool.php(268): QXS\WorkerPool\Semaphore->create()
#1 /mnt/d/dev/Patrickz/test/test.php(59): QXS\WorkerPool\WorkerPool->create()
#2 {main}
thrown in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php on line 99
PHP Fatal error: Uncaught QXS\WorkerPool\SemaphoreException: Semaphore hasn't yet been created. in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php:186
Stack trace:
#0 /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/WorkerPool.php(443): QXS\WorkerPool\Semaphore->destroy()
#1 /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/WorkerPool.php(390): QXS\WorkerPool\WorkerPool->destroy()
#2 [internal function]: QXS\WorkerPool\WorkerPool->onShutDown()
#3 {main}
thrown in /mnt/d/dev/Patrickz/test/vendor/qxsch/worker-pool/src/Semaphore.php on line 186
PHP 8.3.0RC1 (cli) (built: Sep 2 2023 06:34:23) (NTS)
When using automatic respawn a PHP warning is triggered in SimpleSocket.php:66, originating from WorkerPool.php:637:
socket_select(): unable to select [4]: Interrupted system call
Based on this answer from SO, I guess that pcntl_alarm
is interferring with socket_select
.
Will be good if we can catch of the worker exceptions. For example:
$wp = new WorkerPool();
$wp->onWorkerException(function(\Exception $ex, Worker $worker) {
// there we can handle the exception (logging, restart the task or other)
});
Glancing over some of the documentation it's not clear to me what the exact requirements are.
Would be nice to include those in the readme/wiki.
Hi!
When all jobs are completed and the script is about the complete I get several warnings about invalid socket resources.
Warning: socket_last_error(): 420 is not a valid Socket resource /home/[...]/vendor/qxsch/worker-pool/src/SimpleSocket.php 159
Warning: socket_last_error(): 420 is not a valid Socket resource /home/[...]/vendor/qxsch/worker-pool/src/SimpleSocket.php 161
All seems to work properly nonetheless. I run Ubuntu 15 with PHP 5.6.4.
Any ideas on what the origin of those warnings could be?
Many thanks in advance!
Floris
I am using qxsch/WorkerPool.
Thanks to you it is very useful. Thank you.
I am using PHPUnit to test where WorkerPool is used.
When the test is run, the results are not displayed.
Perhaps the execution has stopped and is no longer responding.
% ./vendor/bin/sail exec laravel.test php artisan test
PASS Tests\Unit\ExampleTest
โ that true is true
# (No longer responding)
# (in docker container)
% ps -aux
root 36 0.8 2.6 142896 52908 pts/0 S+ 12:33 0:00 phpunit: Parent
root 37 0.0 1.2 142896 26296 pts/0 S+ 12:33 0:00 phpunit: Worker 1 of QXS\WorkerPool\ClosureWorker [free]
# (No change since then)
A Repository of projects that have been verified to work is temporarily open to the public.
Am I missing something?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.