spatie / fork

A lightweight solution for running code concurrently in PHP

A lightweight solution for running PHP code concurrently

This package makes it easy to run PHP concurrently. Behind the scenes, concurrency is achieved by forking the main PHP process to one or more child tasks.

In this example, where we are going to call an imaginary slow API, all three closures will run at the same time.

use Spatie\Fork\Fork;

$results = Fork::new()
    ->run(
        fn () => (new Api)->fetchData(userId: 1),
        fn () => (new Api)->fetchData(userId: 2),
        fn () => (new Api)->fetchData(userId: 3),
    );

$results[0]; // fetched data of user 1
$results[1]; // fetched data of user 2
$results[2]; // fetched data of user 3

How it works under the hood

✨ In this video on YouTube, we explain how the package works internally.

Support us

We invest a lot of resources into creating best in class open source packages. You can support us by buying one of our paid products.

We highly appreciate you sending us a postcard from your hometown, mentioning which of our package(s) you are using. You'll find our address on our contact page. We publish all received postcards on our virtual postcard wall.

Requirements

This package requires PHP 8 and the pcntl extensions which is installed in many Unix and Mac systems by default.

❗️ pcntl only works in CLI processes, not in a web context. ❗️ posix required for correct handling of process termination for Alpine Linux.

Installation

You can install the package via composer:

composer require spatie/fork

Usage

You can pass as many closures as you want to run. They will be run concurrently. The run function will return an array with the return values of the executed closures.

use Spatie\Fork\Fork;

$results = Fork::new()
    ->run(
        function ()  {
            sleep(1);

            return 'result from task 1';
        },
        function ()  {
            sleep(1);

            return 'result from task 2';
        },
        function ()  {
            sleep(1);

            return 'result from task 3';
        },
    );

// this code will be reached this point after 1 second
$results[0]; // contains 'result from task 1'
$results[1]; // contains 'result from task 2'
$results[2]; // contains 'result from task 3'

Running code before and after each closure

If you need to execute some code before or after each callable passed to run, you can pass a callable to before or after methods. This callable passed will be executed in the child process right before or after the execution of the callable passed to run.

Using before and after in the child task

Here's an example where we are going to get a value from the database using a Laravel Eloquent model. In order to let the child task use the DB, it is necessary to reconnect to the DB. The closure passed to before will run in both child tasks that are created for the closures passed to run.

use App\Models\User;
use Illuminate\Support\Facades\DB;
use Spatie\Fork\Fork;

 Fork::new()
    ->before(fn () => DB::connection('mysql')->reconnect())
    ->run(
        fn () => User::find(1)->someLongRunningFunction(),
        fn () => User::find(2)->someLongRunningFunction(),
    );

If you need to perform some cleanup in the child task after the callable has run, you can use the after method on a Spatie\Fork\Fork instance.

Using before and after in the parent task.

If you need to let the callable passed to before or after run in the parent task, then you need to pass that callable to the parent argument.

use App\Models\User;
use Illuminate\Support\Facades\DB;
use Spatie\Fork\Fork;

 Fork::new()
    ->before(
        parent: fn() => echo 'this runs in the parent task'
    )
    ->run(
        fn () => User::find(1)->someLongRunningFunction(),
        fn () => User::find(2)->someLongRunningFunction(),
    );

You can also pass different closures, to be run in the child and the parent task

use Spatie\Fork\Fork;

Fork::new()
    ->before(
        child: fn() => echo 'this runs in the child task',
        parent: fn() => echo 'this runs in the parent task',
    )
    ->run(
        fn () => User::find(1)->someLongRunningFunction(),
        fn () => User::find(2)->someLongRunningFunction(),
    );

Returning data

All output data is gathered in an array and available as soon as all children are done. In this example, $results will contain three items:

$results = Fork::new()
    ->run(
        fn () => (new Api)->fetchData(userId: 1),
        fn () => (new Api)->fetchData(userId: 2),
        fn () => (new Api)->fetchData(userId: 3),
    );

The output is also available in the after callbacks, which are called whenever a child is done and not at the very end:

$results = Fork::new()
    ->after(
        child: fn (int $i) => echo $i, // 1, 2 and 3
        parent: fn (int $i) => echo $i, // 1, 2 and 3
    )
    ->run(
        fn () => 1,
        fn () => 2,
        fn () => 3,
    );

Finally, return values from child tasks are serialized using PHP's built-in serialize method. This means that you can return anything you can normally serialize in PHP, including objects:

$result = Fork::new()
    ->run(
        fn () => new DateTime('2021-01-01'),
        fn () => new DateTime('2021-01-02'),
    );

Configuring concurrency

By default, all callables will be run in parallel. You can however configure a maximum amount of concurrent processes:

$results = Fork::new()
    ->concurrent(2)
    ->run(
        fn () => 1,
        fn () => 2,
        fn () => 3,
    );

In this case, the first two functions will be run immediately and as soon as one of them finishes, the last one will start as well.

Testing

composer test

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

• Brent Roose
• Freek Van der Herten
• All Contributors

License

The MIT License (MIT). Please see License File for more information.

fork's Issues

All running childs killed one first childs exits

When the first child is done, it seams to call exit, and kill the other childs.

Think it's this exit:

fork/src/Fork.php

Line 132 in ee80755

$this->exit();

that start the killing.

Compatibility with laravel octane Roadrunner

Please is fork compatible with laravel octane running on Roadrunner server?

Fork + RabbitMq

I am using RabbitMq as a queuing system. The problem I have is that after I do
Fork::new() ->run(...$runnable)

the code disconnects from the queuing system. What could be the problem and how can I overcome it? Any suggestions?

using fork with pcntl_signal/pcntl_alarm

When using Fork together with pcntl calls like pcntl_signal (SIGTERM/SIGINT) or pcntl_alarm, I occasionally get the following error:

socket_select(): Unable to select [4]: Interrupted system call

This is a known issue and the common solution is to check for this error and retry again.

[ErrorException] unserialize(): Error at offset 219249 of 219250 bytes

I'm trying to return an key-val array with UTF-8 characters. When running without, it outputs the array fine. Running with 2 files (0.556048 mb) it works fine.

However, when running with 3+ files, 0.78592 mb, I get the unserialize error.

Run Code

// Get Files
$files = Storage::files();

// Returns a key-val array 
$results = Fork::new()
    >run(
         fn() => $this->filesToWordFreq(array_slice($files, 0, 3)),
      );

With the debug below.

   ErrorException 

  unserialize(): Error at offset 219249 of 219250 bytes

  at vendor/spatie/fork/src/Task.php:115
    111▕         $output = $this->output;
    112▕ 
    113▕         if (str_starts_with($output, self::SERIALIZATION_TOKEN)) {
    114▕             $output = unserialize(
  ➜ 115▕                 substr($output, strlen(self::SERIALIZATION_TOKEN))
    116▕             );
    117▕         }
    118▕ 
    119▕         return $output;

I've already increased my php memory limit, with ini_set('memory_limit', '16384M'); earlier.

Is there a hard limit to output size when using spatie/fork? As I was planning to run this with 27k files, not 3 just 3.

EDIT

When dumping the $output in output(), the value returned has been cut off. Does the output concatenation have some form of memory limit?

        foreach ($this->connection->read() as $output) {
            $this->output .= $output;
        }

Dumping the variable in execute() has the entire string, so something goes wrong after the execute() function.

Undefined array key 51266

Undefined array key 51266
vendor/laravel/framework/src/Illuminate/Foundation/Console/ServeCommand.php
or this ,
Undefined array key 37790
at vendor/laravel/framework/src/Illuminate/Foundation/Console/ServeCommand.php :
i get this errors when i try to excute from get api request but when i make a laravel consol command its works fine why is it ?

Terminating a process after its execution does not work correctly

After PHP 8.2 exit no longer works as a graceful process termination.

I tried to solve the problem in this way by replacing exit with posix_kill(posix_getpid(), SIGKILL); and it started working correctly, but I'm not sure if the result is correct.

This code will perform only 2 tasks and will hang indefinitely waiting for the end of the processes.

Example:

$results = Fork::new()
    ->concurrent(2)
    ->run(
        fn () => 1,
        fn () => 2,
        fn () => 3,
    );

Laravel: 10.10.1
PHP: 8.2.6
OS (Docker): Alpine (3.18.0)

Processes are being terminated unexpectedly

I'm not sure if I'm doing something wrong or if it's a bug, but when I run the code below, more often than not one or more tasks don't finish.

Fork::new()
    ->run(
        function () {
            foreach (range(1, 1000) as $i) {
                usleep(rand(300, 3000));
            }

            echo "## Done #1" . PHP_EOL;
        },
        function () {
            foreach (range(1, 1000) as $i) {
                usleep(rand(300, 3000));
            }

            echo "## Done #2" . PHP_EOL;
        },
        function () {
            foreach (range(1, 1000) as $i) {
                usleep(rand(300, 3000));
            }

            echo "## Done #3" . PHP_EOL;
        },
    );

I'm running on a MacBook with M2 chip using Docker, below the Dockerfile I run the tests:

FROM php:8.2-cli-bookworm as base

# Install system dependencies
RUN set -e \
  && apt update \
  && apt install -y \
    git \
    unzip \
    libzip-dev \
    procps \
    autoconf \
    automake \
    gcc \
    make \
    g++ \
    coreutils \
    libtool \
    libssl-dev \
    libpq-dev \
    libicu-dev \
    libpcre3-dev \
    zlib1g-dev

# Install igbinary extension
RUN pecl install -o -f igbinary \
  && rm -rf /tmp/pear \
  && docker-php-ext-enable igbinary

# Install Redis extension with igbinary support
RUN cd /tmp && git clone https://github.com/phpredis/phpredis.git \
    && cd phpredis \
    && phpize \
    && ./configure --enable-redis-igbinary \
    && make && make install \
    && docker-php-ext-enable redis

# Install PHP extensions
RUN docker-php-ext-install \
    intl \
    posix \
    sockets \
    pcntl \
    zip

# Install Composer globally
RUN curl https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

# Copy PHP configuration files
COPY docker/php/config/common/* /usr/local/etc/php/conf.d/

# Add source code into docker image
COPY . /app

WORKDIR /app

Bug with unserialize big json from api with 200 items

I'm using the following code:

use Curl\Curl;
use Spatie\Fork\Fork;

$output = Fork::new()->run(
    function () {
        $curl = new Curl();
        return $curl->get('https://jsonplaceholder.typicode.com/todos');
    }
);

var_dump($output);

When getting all the todo's from the api it will throw an exception:
Notice: unserialize(): Error at offset 8178 of 8178 bytes .... /src/Task.php line 115

I already have increased the buffer size(tried increasing until the max) but still got this problem/bug.
When i slice the array of items to 60 instead of 200 that come from that response.

Is there a solution for this problem/bug?

Endless loop outside of tests

Sorry to bother you guys. I'm actively looking for a fix for this and trying a bunch of stuff but a problem shared...

So, I'm using a MacBook Pro 2020 Intel variant. I run Laravel locally using Valet. My test suite is obviously using PHP to run as you'd expect. I'm on the latest version of valet and I'm using PHP 8.0.5.

When I run my test suite, I get the expected result:

$test = Fork::new()->run(fn() => 'Hello', fn() => 'World');
dd($test);

So far, so breezy. However, if I switch to the browser and hit a route that calls this code, the dd never takes place.

I've done some debugging (mainly by placing ray calls at various points in the Fork class), and the issue seems to stem from the call to pcntl_waitpid in Task on line 133. For whatever reason, in the browser, this always returns a status code of 0. The output on line 131 gets set correctly, but it still doesn't return the PID to say it has completed the closure.

Thanks for all your hard work! I'll keep looking into this from my end.