GithubHelp home page GithubHelp logo

swoole / phpkafka Goto Github PK

View Code? Open in Web Editor NEW
257.0 257.0 45.0 497 KB

PHP Kafka client is used in PHP-FPM and Swoole. PHP Kafka client supports 50 APIs, which might be one that supports the most message types ever.

Home Page: https://longlang.org/

License: Apache License 2.0

PHP 100.00%
kafka php swoole

phpkafka's Introduction

Swoole Logo
Swoole is an event-driven, asynchronous, coroutine-based concurrency library with high performance for PHP.

lib-swoole ext-swoole test-linux Frameworks Tests codecov

Twitter Discord Latest Release License Coverity Scan Build Status

⚙️ Quick Start

Run Swoole program by Docker

docker run --rm phpswoole/swoole "php --ri swoole"

For details on how to use it, see: How to Use This Image.

Documentation

https://wiki.swoole.com/

HTTP Service

$http = new Swoole\Http\Server('127.0.0.1', 9501);
$http->set(['hook_flags' => SWOOLE_HOOK_ALL]);

$http->on('request', function ($request, $response) {
    $result = [];
    Co::join([
        go(function () use (&$result) {
            $result['google'] = file_get_contents("https://www.google.com/");
        }),
        go(function () use (&$result) {
            $result['taobao'] = file_get_contents("https://www.taobao.com/");
        })
    ]);
    $response->end(json_encode($result));
});

$http->start();

Concurrency

Co\run(function() {
    Co\go(function() {
        while(1) {
            sleep(1);
            $fp = stream_socket_client("tcp://127.0.0.1:8000", $errno, $errstr, 30);
            echo fread($fp, 8192), PHP_EOL;
        }
    });

    Co\go(function() {
        $fp = stream_socket_server("tcp://0.0.0.0:8000", $errno, $errstr, STREAM_SERVER_BIND | STREAM_SERVER_LISTEN);
        while(1) {
            $conn = stream_socket_accept($fp);
            fwrite($conn, 'The local time is ' . date('n/j/Y g:i a'));
        }
    });

    Co\go(function() {
        $redis = new Redis();
        $redis->connect('127.0.0.1', 6379);
        while(true) {
            $redis->subscribe(['test'], function ($instance, $channelName, $message) {
                echo 'New redis message: '.$channelName, "==>", $message, PHP_EOL;
            });
        }
    });

    Co\go(function() {
        $redis = new Redis();
        $redis->connect('127.0.0.1', 6379);
        $count = 0;
        while(true) {
            sleep(2);
            $redis->publish('test','hello, world, count='.$count++);
        }
    });
});

Runtime Hook

Swoole hooks the blocking io function of PHP at the bottom layer and automatically converts it to a non-blocking function, so that these functions can be called concurrently in coroutines.

Supported extension/functions

  • ext-curl (Support symfony and guzzle)
  • ext-redis
  • ext-mysqli
  • ext-pdo_mysql
  • ext-pdo_pgsql
  • ext-pdo_sqlite
  • ext-pdo_oracle
  • ext-pdo_odbc
  • stream functions (e.g. stream_socket_client/stream_socket_server), Supports TCP/UDP/UDG/Unix/SSL/TLS/FileSystem API/Pipe
  • ext-sockets
  • ext-soap
  • sleep/usleep/time_sleep_until
  • proc_open
  • gethostbyname/shell_exec/exec
  • fread/fopen/fsockopen/fwrite/flock

🛠 Develop & Discussion

💎 Awesome Swoole

Project Awesome Swoole maintains a curated list of awesome things related to Swoole, including

  • Swoole-based frameworks and libraries.
  • Packages to integrate Swoole with popular PHP frameworks, including Laravel, Symfony, Slim, and Yii.
  • Books, videos, and other learning materials about Swoole.
  • Debugging, profiling, and testing tools for developing Swoole-based applications.
  • Coroutine-friendly packages and libraries.
  • Other Swoole related projects and resources.

✨ Event-based

The network layer in Swoole is event-based and takes full advantage of the underlying epoll/kqueue implementation, making it really easy to serve millions of requests.

Swoole 4.x uses a brand new engine kernel and now it has a full-time developer team, so we are entering an unprecedented period in PHP history which offers a unique possibility for rapid evolution in performance.

⚡ Coroutine

Swoole 4.x or later supports the built-in coroutine with high availability, and you can use fully synchronized code to implement asynchronous performance. PHP code without any additional keywords, the underlying automatic coroutine-scheduling.

Developers can understand coroutines as ultra-lightweight threads, and you can easily create thousands of coroutines in a single process.

MySQL

Concurrency 10K requests to read data from MySQL takes only 0.2s!

$s = microtime(true);
Co\run(function() {
    for ($c = 100; $c--;) {
        go(function () {
            $mysql = new Swoole\Coroutine\MySQL;
            $mysql->connect([
                'host' => '127.0.0.1',
                'user' => 'root',
                'password' => 'root',
                'database' => 'test'
            ]);
            $statement = $mysql->prepare('SELECT * FROM `user`');
            for ($n = 100; $n--;) {
                $result = $statement->execute();
                assert(count($result) > 0);
            }
        });
    }
});
echo 'use ' . (microtime(true) - $s) . ' s';

Mixed server

You can create multiple services on the single event loop: TCP, HTTP, Websocket and HTTP2, and easily handle thousands of requests.

function tcp_pack(string $data): string
{
    return pack('N', strlen($data)) . $data;
}
function tcp_unpack(string $data): string
{
    return substr($data, 4, unpack('N', substr($data, 0, 4))[1]);
}
$tcp_options = [
    'open_length_check' => true,
    'package_length_type' => 'N',
    'package_length_offset' => 0,
    'package_body_offset' => 4
];
$server = new Swoole\WebSocket\Server('127.0.0.1', 9501, SWOOLE_BASE);
$server->set(['open_http2_protocol' => true]);
// http && http2
$server->on('request', function (Swoole\Http\Request $request, Swoole\Http\Response $response) {
    $response->end('Hello ' . $request->rawcontent());
});
// websocket
$server->on('message', function (Swoole\WebSocket\Server $server, Swoole\WebSocket\Frame $frame) {
    $server->push($frame->fd, 'Hello ' . $frame->data);
});
// tcp
$tcp_server = $server->listen('127.0.0.1', 9502, SWOOLE_TCP);
$tcp_server->set($tcp_options);
$tcp_server->on('receive', function (Swoole\Server $server, int $fd, int $reactor_id, string $data) {
    $server->send($fd, tcp_pack('Hello ' . tcp_unpack($data)));
});
$server->start();

Coroutine clients

Whether you DNS query or send requests or receive responses, all of these are scheduled by coroutine automatically.

go(function () {
    // http
    $http_client = new Swoole\Coroutine\Http\Client('127.0.0.1', 9501);
    assert($http_client->post('/', 'Swoole Http'));
    var_dump($http_client->body);
    // websocket
    $http_client->upgrade('/');
    $http_client->push('Swoole Websocket');
    var_dump($http_client->recv()->data);
});
go(function () {
    // http2
    $http2_client = new Swoole\Coroutine\Http2\Client('localhost', 9501);
    $http2_client->connect();
    $http2_request = new Swoole\Http2\Request;
    $http2_request->method = 'POST';
    $http2_request->data = 'Swoole Http2';
    $http2_client->send($http2_request);
    $http2_response = $http2_client->recv();
    var_dump($http2_response->data);
});
go(function () use ($tcp_options) {
    // tcp
    $tcp_client = new Swoole\Coroutine\Client(SWOOLE_TCP);
    $tcp_client->set($tcp_options);
    $tcp_client->connect('127.0.0.1', 9502);
    $tcp_client->send(tcp_pack('Swoole Tcp'));
    var_dump(tcp_unpack($tcp_client->recv()));
});

Channel

Channel is the only way for exchanging data between coroutines, the development combination of the Coroutine + Channel is the famous CSP programming model.

In Swoole development, Channel is usually used for implementing connection pool or scheduling coroutine concurrent.

The simplest example of a connection pool

In the following example, we have a thousand concurrently requests to redis. Normally, this has exceeded the maximum number of Redis connections setting and will throw a connection exception, but the connection pool based on Channel can perfectly schedule requests. We don't have to worry about connection overload.

class RedisPool
{
    /**@var \Swoole\Coroutine\Channel */
    protected $pool;

    /**
     * RedisPool constructor.
     * @param int $size max connections
     */
    public function __construct(int $size = 100)
    {
        $this->pool = new \Swoole\Coroutine\Channel($size);
        for ($i = 0; $i < $size; $i++) {
            $redis = new \Swoole\Coroutine\Redis();
            $res = $redis->connect('127.0.0.1', 6379);
            if ($res == false) {
                throw new \RuntimeException("failed to connect redis server.");
            } else {
                $this->put($redis);
            }
        }
    }

    public function get(): \Swoole\Coroutine\Redis
    {
        return $this->pool->pop();
    }

    public function put(\Swoole\Coroutine\Redis $redis)
    {
        $this->pool->push($redis);
    }

    public function close(): void
    {
        $this->pool->close();
        $this->pool = null;
    }
}

go(function () {
    $pool = new RedisPool();
    // max concurrency num is more than max connections
    // but it's no problem, channel will help you with scheduling
    for ($c = 0; $c < 1000; $c++) {
        go(function () use ($pool, $c) {
            for ($n = 0; $n < 100; $n++) {
                $redis = $pool->get();
                assert($redis->set("awesome-{$c}-{$n}", 'swoole'));
                assert($redis->get("awesome-{$c}-{$n}") === 'swoole');
                assert($redis->delete("awesome-{$c}-{$n}"));
                $pool->put($redis);
            }
        });
    }
});

Producer and consumers

Some Swoole's clients implement the defer mode for concurrency, but you can still implement it flexible with a combination of coroutines and channels.

go(function () {
    // User: I need you to bring me some information back.
    // Channel: OK! I will be responsible for scheduling.
    $channel = new Swoole\Coroutine\Channel;
    go(function () use ($channel) {
        // Coroutine A: Ok! I will show you the github addr info
        $addr_info = Co::getaddrinfo('github.com');
        $channel->push(['A', json_encode($addr_info, JSON_PRETTY_PRINT)]);
    });
    go(function () use ($channel) {
        // Coroutine B: Ok! I will show you what your code look like
        $mirror = Co::readFile(__FILE__);
        $channel->push(['B', $mirror]);
    });
    go(function () use ($channel) {
        // Coroutine C: Ok! I will show you the date
        $channel->push(['C', date(DATE_W3C)]);
    });
    for ($i = 3; $i--;) {
        list($id, $data) = $channel->pop();
        echo "From {$id}:\n {$data}\n";
    }
    // User: Amazing, I got every information at earliest time!
});

Timer

$id = Swoole\Timer::tick(100, function () {
    echo "⚙️ Do something...\n";
});
Swoole\Timer::after(500, function () use ($id) {
    Swoole\Timer::clear($id);
    echo "⏰ Done\n";
});
Swoole\Timer::after(1000, function () use ($id) {
    if (!Swoole\Timer::exists($id)) {
        echo "✅ All right!\n";
    }
});

The way of coroutine

go(function () {
    $i = 0;
    while (true) {
        Co::sleep(0.1);
        echo "📝 Do something...\n";
        if (++$i === 5) {
            echo "🛎 Done\n";
            break;
        }
    }
    echo "🎉 All right!\n";
});

🔥 Amazing runtime hooks

As of Swoole v4.1.0, we added the ability to transform synchronous PHP network libraries into co-routine libraries using a single line of code.

Simply call the Swoole\Runtime::enableCoroutine() method at the top of your script. In the sample below we connect to php-redis and concurrently read 10k requests in 0.1s:

Swoole\Runtime::enableCoroutine();
$s = microtime(true);
Co\run(function() {
    for ($c = 100; $c--;) {
        go(function () {
            ($redis = new Redis)->connect('127.0.0.1', 6379);
            for ($n = 100; $n--;) {
                assert($redis->get('awesome') === 'swoole');
            }
        });
    }
});
echo 'use ' . (microtime(true) - $s) . ' s';

By calling this method, the Swoole kernel replaces ZendVM stream function pointers. If you use php_stream based extensions, all socket operations can be dynamically converted to be asynchronous IO scheduled by coroutine at runtime!

How many things you can do in 1s?

Sleep 10K times, read, write, check and delete files 10K times, use PDO and MySQLi to communicate with the database 10K times, create a TCP server and multiple clients to communicate with each other 10K times, create a UDP server and multiple clients to communicate with each other 10K times... Everything works well in one process!

Just see what the Swoole brings, just imagine...

Swoole\Runtime::enableCoroutine();
$s = microtime(true);
Co\run(function() {
    // i just want to sleep...
    for ($c = 100; $c--;) {
        go(function () {
            for ($n = 100; $n--;) {
                usleep(1000);
            }
        });
    }

    // 10K file read and write
    for ($c = 100; $c--;) {
        go(function () use ($c) {
            $tmp_filename = "/tmp/test-{$c}.php";
            for ($n = 100; $n--;) {
                $self = file_get_contents(__FILE__);
                file_put_contents($tmp_filename, $self);
                assert(file_get_contents($tmp_filename) === $self);
            }
            unlink($tmp_filename);
        });
    }

    // 10K pdo and mysqli read
    for ($c = 50; $c--;) {
        go(function () {
            $pdo = new PDO('mysql:host=127.0.0.1;dbname=test;charset=utf8', 'root', 'root');
            $statement = $pdo->prepare('SELECT * FROM `user`');
            for ($n = 100; $n--;) {
                $statement->execute();
                assert(count($statement->fetchAll()) > 0);
            }
        });
    }
    for ($c = 50; $c--;) {
        go(function () {
            $mysqli = new Mysqli('127.0.0.1', 'root', 'root', 'test');
            $statement = $mysqli->prepare('SELECT `id` FROM `user`');
            for ($n = 100; $n--;) {
                $statement->bind_result($id);
                $statement->execute();
                $statement->fetch();
                assert($id > 0);
            }
        });
    }

    // php_stream tcp server & client with 12.8K requests in single process
    function tcp_pack(string $data): string
    {
        return pack('n', strlen($data)) . $data;
    }

    function tcp_length(string $head): int
    {
        return unpack('n', $head)[1];
    }

    go(function () {
        $ctx = stream_context_create(['socket' => ['so_reuseaddr' => true, 'backlog' => 128]]);
        $socket = stream_socket_server(
            'tcp://0.0.0.0:9502',
            $errno, $errstr, STREAM_SERVER_BIND | STREAM_SERVER_LISTEN, $ctx
        );
        if (!$socket) {
            echo "$errstr ($errno)\n";
        } else {
            $i = 0;
            while ($conn = stream_socket_accept($socket, 1)) {
                stream_set_timeout($conn, 5);
                for ($n = 100; $n--;) {
                    $data = fread($conn, tcp_length(fread($conn, 2)));
                    assert($data === "Hello Swoole Server #{$n}!");
                    fwrite($conn, tcp_pack("Hello Swoole Client #{$n}!"));
                }
                if (++$i === 128) {
                    fclose($socket);
                    break;
                }
            }
        }
    });
    for ($c = 128; $c--;) {
        go(function () {
            $fp = stream_socket_client("tcp://127.0.0.1:9502", $errno, $errstr, 1);
            if (!$fp) {
                echo "$errstr ($errno)\n";
            } else {
                stream_set_timeout($fp, 5);
                for ($n = 100; $n--;) {
                    fwrite($fp, tcp_pack("Hello Swoole Server #{$n}!"));
                    $data = fread($fp, tcp_length(fread($fp, 2)));
                    assert($data === "Hello Swoole Client #{$n}!");
                }
                fclose($fp);
            }
        });
    }

    // udp server & client with 12.8K requests in single process
    go(function () {
        $socket = new Swoole\Coroutine\Socket(AF_INET, SOCK_DGRAM, 0);
        $socket->bind('127.0.0.1', 9503);
        $client_map = [];
        for ($c = 128; $c--;) {
            for ($n = 0; $n < 100; $n++) {
                $recv = $socket->recvfrom($peer);
                $client_uid = "{$peer['address']}:{$peer['port']}";
                $id = $client_map[$client_uid] = ($client_map[$client_uid] ?? -1) + 1;
                assert($recv === "Client: Hello #{$id}!");
                $socket->sendto($peer['address'], $peer['port'], "Server: Hello #{$id}!");
            }
        }
        $socket->close();
    });
    for ($c = 128; $c--;) {
        go(function () {
            $fp = stream_socket_client("udp://127.0.0.1:9503", $errno, $errstr, 1);
            if (!$fp) {
                echo "$errstr ($errno)\n";
            } else {
                for ($n = 0; $n < 100; $n++) {
                    fwrite($fp, "Client: Hello #{$n}!");
                    $recv = fread($fp, 1024);
                    list($address, $port) = explode(':', (stream_socket_get_name($fp, true)));
                    assert($address === '127.0.0.1' && (int)$port === 9503);
                    assert($recv === "Server: Hello #{$n}!");
                }
                fclose($fp);
            }
        });
    }
});
echo 'use ' . (microtime(true) - $s) . ' s';

⌛️ Installation

As with any open source project, Swoole always provides the most reliable stability and the most powerful features in the latest released version. Please ensure as much as possible that you are using the latest version.

Compiling requirements

  • Linux, OS X or Cygwin, WSL
  • PHP 7.2.0 or later (The higher the version, the better the performance.)
  • GCC 4.8 or later

1. Install via PECL (beginners)

pecl install swoole

2. Install from source (recommended)

Please download the source packages from Releases or:

git clone https://github.com/swoole/swoole-src.git && \
cd swoole-src

Compile and install at the source folder:

phpize && \
./configure && \
make && make install

Enable extension in PHP

After compiling and installing to the system successfully, you have to add a new line extension=swoole.so to php.ini to enable Swoole extension.

Extra compiler configurations

for example: ./configure --enable-openssl --enable-sockets

  • --enable-openssl or --with-openssl-dir=DIR
  • --enable-sockets
  • --enable-mysqlnd (need mysqlnd, it just for supporting $mysql->escape method)
  • --enable-swoole-curl

Upgrade

⚠️ If you upgrade from source, don't forget to make clean before you upgrade your swoole

  1. pecl upgrade swoole
  2. cd swoole-src && git pull && make clean && make && sudo make install
  3. if you change your PHP version, please re-run phpize clean && phpize then try to compile

Major change since version 4.3.0

Async clients and API are moved to a separate PHP extension swoole_async since version 4.3.0, install swoole_async:

git clone https://github.com/swoole/ext-async.git
cd ext-async
phpize
./configure
make -j 4
sudo make install

Enable it by adding a new line extension=swoole_async.so to php.ini.

🍭 Benchmark

  • On the open source Techempower Web Framework benchmarks Swoole used MySQL database benchmark to rank first, and all performance tests ranked in the first echelon.
  • You can just run Benchmark Script to quickly test the maximum QPS of Swoole-HTTP-Server on your machine.

🔰️ Security issues

Security issues should be reported privately, via email, to the Swoole develop team [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message.

🖊️ Contribution

Your contribution to Swoole development is very welcome!

You may contribute in the following ways:

❤️ Contributors

This project exists thanks to all the people who contribute. [Contributors].

🎙️ Official Evangelist

Demin has been playing with PHP since 2000, focusing on building high-performance, secure web services. He is an occasional conference speaker on PHP and Swoole, and has been working for companies in the states like eBay, Visa and Glu Mobile for years. You may find Demin on Twitter or GitHub.

📃 License

Apache License Version 2.0 see http://www.apache.org/licenses/LICENSE-2.0.html

phpkafka's People

Contributors

dygin avatar huangdijia avatar leocavalcante avatar limingxinleo avatar pandaliu-1111 avatar qiqizjl avatar reasno avatar reneeteng avatar sy-records avatar szutoutou avatar yurunsoft avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

phpkafka's Issues

Acks设置为0时,会重复插入4条完全相同的数据

  • 你遇到了什么问题?

Acks设置为0时,会重复插入4条完全相同的数据

  • Kafka 环境是自建还是云服务?

自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

PHP 7.4.16 (cli) (built: Mar  5 2021 07:54:20) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.16, Copyright (c), by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.6.7
Built => Jun  1 2021 10:51:44
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.1.1g  21 Apr 2020
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608

  • 提供最小可复现代码:
https://github.com/longyan/phpkafka/blob/master/examples/producer.php    

$config->setAcks(0);   //修改成0即可 

个人觉得 应该是 vendor/longlang/phpkafka/src/Producer/Producer.php 152行 continue; 的问题 ,应该改为 break;

请问什么原因导致了fetchMessages 中 $currentList 为空

[ERROR] App\Exception\Handler\ErrorHandler: Invalid argument supplied for foreach()(2) in /Users/zhaoxingyan/data/hives-hyperf/vendor/longlang/phpkafka/src/Consumer/Consumer.php:293
Stack trace:
#0 /Users/zhaoxingyan/data/hives-hyperf/vendor/longlang/phpkafka/src/Consumer/Consumer.php(293): ErrorHandler(2, 'Invalid argumen...', '/Users/zhaoxing...', 293, Array)
#1 /Users/zhaoxingyan/data/hives-hyperf/vendor/longlang/phpkafka/src/Consumer/Consumer.php(225): longlang\phpkafka\Consumer\Consumer->fetchMessages()
#2 /Users/zhaoxingyan/data/hives-hyperf/vendor/longlang/phpkafka/src/Consumer/Consumer.php(203): longlang\phpkafka\Consumer\Consumer->consume()
#3 /Users/zhaoxingyan/data/hives-hyperf/vendor/hyperf/kafka/src/ConsumerManager.php(150): longlang\phpkafka\Consumer\Consumer->start()
#4 /Users/zhaoxingyan/data/hives-hyperf/vendor/hyperf/utils/src/Functions.php(80): class@anonymous->Hyperf\Kafka{closure}()
#5 /Users/zhaoxingyan/data/hives-hyperf/vendor/hyperf/kafka/src/ConsumerManager.php(157): retry(0, Object(Closure), 10)
#6 /Users/zhaoxingyan/data/hives-hyperf/vendor/hyperf/process/src/AbstractProcess.php(134): class@anonymous->handle()

消费进程执行一段时间后就出错 重启

  • 你遇到了什么问题?

image

  • Kafka 环境是自建还是云服务?

阿里云 kafka 服务

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

bash-5.0# php -v & php --ri swoole & composer info | grep longlang/phpkafka
[1] 166
[2] 167
PHP 8.0.13 (cli) (built: Nov 20 2021 07:13:05) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.13, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.13, Copyright (c), by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.8.9
Built => Apr 17 2022 02:42:36
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.1.1n  15 Mar 2022
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.12
mutex_timedlock => enabled
pthread_barrier => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608
longlang/phpkafka                v1.2.1             A kafka client. Support php-fpm and Swoole.
[1]-  Done                    php -v
[2]+  Done                    php --ri swoole
  • 提供最小可复现代码:
<?php

declare(strict_types=1);

namespace App\kafka;

use Hyperf\Kafka\AbstractConsumer;
use Hyperf\Kafka\Annotation\Consumer;
use longlang\phpkafka\Consumer\ConsumeMessage;

/**
 * @Consumer(topic="hyperf", nums=5, groupId="hyperf", autoCommit=true)
 */
class KafkaConsumer extends AbstractConsumer
{
    public function consume(ConsumeMessage $message): string
    {
        var_dump($message->getTopic() . ':' . $message->getKey() . ':' . $message->getValue());
    }
}

a new process run Consumer client show out of memory

php 7.2

swoole 4.4.16

kafka 7.8.1

swoft 2.0.8

a new process run Consumer client

====================================================================
public function run(Process $process): void
{

     Co::create(function(){

         $config = new ConsumerConfig();
         $config->setBroker('127.0.0.1:9092');
         $config->setTopic('ordertest');
         $consumer = new Consumer($config);

         while (true) {
             $message = $consumer->consume();
             if($message)
             {
                 CLog::info($message->getKey() . ':' . $message->getValue());
                 $consumer->ack($message->getPartition()); // ack
             }
         }
     });
}

=====================================================================

error:
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /home/www/swoft/vendor/longlang/phpkafka/src/Protocol/Type/ArrayInt32.php on line 60
PHP Fatal error: Uncaught ErrorException: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /home/www/swoft/vendor/longlang/phpkafka/src/Protocol/Type/ArrayInt32.php:60
Stack trace:

=====================================================================

Sasl 相关的代码现在composer的包无法拉取到

  • 你遇到了什么问题?
    Sasl 相关的代码现在composer的包无法拉取到 是不是没发版

  • Kafka 环境是自建还是云服务?

阿里云

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里

  • 提供最小可复现代码:
// 你的代码

报错信息

{"message":"longlang\phpkafka\Consumer\Consumer::fetchMessages","context":[{"class":"RuntimeException","message":"Partition 11 doses not exists","code":0,"file":"vendor/longlang/phpkafka/src/Consumer/OffsetManager.php:142"}],"level":200,"level_name":"INFO","channel":"default","datetime":"2021-04-01T21:28:27.426031+08:00","extra":{"host":","traceid":"afeadb577ff94de3cc0c7af6f8196c24e858eaae"}}

PHP 7.4.12 (cli) (built: Dec 30 2020 11:56:19) ( NTS DEBUG )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

Int32::unpack() must be of the type int, null returned

  • 你遇到了什么问题?

Fatal error: Uncaught TypeError: Return value of longlang\phpkafka\Protocol\Type\Int32::unpack() must be of the type int, null returned in /var/www/html/live-php/vendor/longlang/phpkafka/src/Protocol/Type/Int32.php:43

  • Kafka 环境是自建还是云服务?

腾讯云 ckafka 0.10.2.1

  • 请执行下面的命令获取环境信息。

php -v & composer info | grep longlang/phpkafka

PHP 7.4.28 (cli) (built: Mar 18 2022 01:43:55) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.28, Copyright (c), by Zend Technologies
longlang/phpkafka                  dev-master f154746 A kafka client. Support php-fpm and Swoole.
  • 提供最小可复现代码:
produce.php

关于消费者使用建议

当前在调用消费者对象时需要手动指定消费的partition
$config->setPartitions([0,1,2]);
在有多个consumer进程时会有 [27] The group is rebalancing, so a rejoin is needed 异常。
期望可以改进consumer方法,不需要使用者人工指定partition, 如果是个多进程的消费方式就显得不太友好,特别是分布式部署时。
希望可以自动均衡分配consumer group对象的partition。

配置新起的消费者组配置从头消费还是从最新的消息开始消费

  • 你遇到了什么问题?

在新起的消费者组如果在broker端没有对应的消费位移的话,默认从offset为0开始消费,但分区前面的数据可能过期被删除了 对应的offset可能不是从0开始的 fetchMessage时候会报out_of_range,这时候看代码里获取了一次分区提交的最新偏移量从最新的消息开始消费。这样不能自己配置在新起消费者后是从头消费还是从最新的消息开始消费。

后期是否可以加一个功能,java版的是auto.offset.reset 对应的配置是earliest/latest,如果在broker里找不到对应的消费位移的话 就会根据这个配置 去取分区最早的偏移量或者最新的偏移量,这样可以在新起了消费者组后配置是从头开始消费还是从最新的消息开始消费。

  • Kafka 环境是自建还是云服务?

自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里

PHP 7.4.19 (cli) (built: May 31 2021 04:55:16) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Yasd v0.3.8, Our Copyright, by codinghuang

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.4.15
Built => May 31 2021 05:59:33
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.0.2k-fips  26 Jan 2017
http2 => enabled
pcre => enabled
zlib => 1.2.7
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? yes
longlang/phpkafka                  v1.1.4  A kafka client. Support php-fpm and Swoole.

  • 提供最小可复现代码:
// 你的代码

当OFFSET_OUT_OF_RANGE时,调用ListOffsetRequest时 currentLeaderEpoch没有设定,导致响应的数据是错的

  • 你遇到了什么问题?

当OFFSET_OUT_OF_RANGE时,调用ListOffsetRequest时 currentLeaderEpoch没有设定,导致响应的数据是错的

  • Kafka 环境是自建还是云服务?

自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里
[1] 26314
[2] 26315

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.4.15
Built => May 31 2021 05:59:33
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.0.2k-fips  26 Jan 2017
http2 => enabled
pcre => enabled
zlib => 1.2.7
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
PHP 7.4.19 (cli) (built: May 31 2021 04:55:16) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Yasd v0.3.8, Our Copyright, by codinghuang
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? 
longlang/phpkafka                  v1.1.4  A kafka client. Support php-fpm and Swoole.

  • 提供最小可复现代码:
// 你的代码
ListOffsetPartition类的$currentLeaderEpoch属性默认是0 ,请求是没有指定,返回的offset是错误的
![image](https://user-images.githubusercontent.com/19905508/127950842-92699354-e950-4884-b185-1d8c9464de3d.png)

//在OffsetManager的updateListOffsets方法里指定后 返回是正常的:
 $listOffsetPartition->setPartitionIndex($partition)->setTimestamp(-1)->setCurrentLeaderEpoch($leaderEpoch[$partition]);
![image](https://user-images.githubusercontent.com/19905508/127950974-1a408f6e-49fa-4426-a58e-a2ab5ca3136f.png)

minBytes和maxBytes未生效

设置的minBytes为1024 * 1024 maxBytes为5 * 1024 * 1024,但每次fetch后响应的records里都只有1条记录。
详情看下边截图,kafka中的每条消息都只有十几字节
image

image

Authentication using the SCRAM-SHA-512 mechanism

  • What problem did you encounter?

Can you show an example of authentication using the SCRAM-SHA-512 mechanism? (with SASL_SSL)

An example from the cloud service documentation looks like this

kafkacat -C \
          -b adress:9091 \
          -t topic \
          -X security.protocol=SASL_SSL\
          -X sasl.mechanisms=SCRAM-SHA-512 \
          -X sasl.username="user login" \
          -X sasl.password="consumer password" \
          -X ssl.ca.location=/usr/local/share/ca-certificates/Yandex/YandexCA.crt -Z -K:
  • Is the Kafka environment self-built or cloud service?

cloud service

  • Please execute the following command to get environment information.

php -v & php --ri swoole & composer info | grep longlang/phpkafka

PHP 7.4.30 (cli) (built: Jun 7 2022 08:38:19) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
with Zend OPcache v7.4.30, Copyright (c), by Zend Technologies
longlang/phpkafka dev-master f154746 A kafka client. Support php-fpm and Swoole.

在swoole服务中, kafka生产数据报错

  • 你遇到了什么问题?
    在swoole服务中, kafka生产数据报错

  • Kafka 环境是自建还是云服务?
    云服务, 版本1.1.1

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里
[1] 25961
PHP 7.2.21 (cli) (built: Aug 13 2019 13:30:58) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.21, Copyright (c) 1999-2018, by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.4.3
Built => Aug 13 2019 14:26:13
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
http2 => enabled
pcre => enabled
zlib => enabled
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
[1]+  Done                    /usr/bin/php72 -v

longlang/phpkafka                  v1.2.0          A kafka client. Support php-fpm and Swoole.
  • 提供最小可复现代码:
# 生产数据就这一个方法
public function producerForTimeOutNotLimit($value)
{
    if (empty($value) || !is_array($value)) throw new Exception('生产数据类型不可为空并且类型为数组');
    $value = json_encode($value);
    $config = new \longlang\phpkafka\Producer\ProducerConfig();
    $config->setBootstrapServer($this->config['service']);
    $config->setConnectTimeout(-1);
    $config->setUpdateBrokers(true);
    $producer = new \longlang\phpkafka\Producer\Producer($config);
    $producer->send($this->config['topic'], $value, '');
}
  • 堆栈信息
[2021-09-26 11:33:13] dev.ERROR: Symfony\Component\Debug\Exception\FatalErrorException: Uncaught longlang\phpkafka\Exception\SocketException: Could not recv data from stream,  [0] in /data/wwwroot/abs/abs_gray3/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php:128
Stack trace:
#0 /data/wwwroot/abs/abs_gray3/vendor/longlang/phpkafka/src/Client/SwooleClient.php(135): longlang\phpkafka\Socket\SwooleSocket->recv(4, -1)
#1 {main}
  thrown in /data/wwwroot/abs/abs_gray3/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php:128
Stack trace:
#0 /data/wwwroot/abs/abs_gray3/vendor/laravel/lumen-framework/src/Concerns/RegistersExceptionHandlers.php(54): Laravel\Lumen\Application->handleShutdown()
#1 [internal function]: Laravel\Lumen\Application->Laravel\Lumen\Concerns\{closure}()
#2 {main} {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\FatalErrorException(code: 1): Uncaught longlang\\phpkafka\\Exception\\SocketException: Could not recv data from stream,  [0] in /data/wwwroot/abs/abs_gray3/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php:128
Stack trace:
#0 /data/wwwroot/abs/abs_gray3/vendor/longlang/phpkafka/src/Client/SwooleClient.php(135): longlang\\phpkafka\\Socket\\SwooleSocket->recv(4, -1)
#1 {main}
  thrown at /data/wwwroot/abs/abs_gray3/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php:128)
[stacktrace]
#0 /data/wwwroot/abs/abs_gray3/vendor/laravel/lumen-framework/src/Concerns/RegistersExceptionHandlers.php(54): Laravel\\Lumen\\Application->handleShutdown()
#1 [internal function]: Laravel\\Lumen\\Application->Laravel\\Lumen\\Concerns\\{closure}()
#2 {main}
"}

不能获取到Header

  • 你遇到了什么问题?

不能获取到Header

  • Kafka 环境是自建还是云服务?

自建

通过工具查询,头部已经设置上了

image

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

[1] 1067
[2] 1068
PHP 8.0.6 (cli) (built: Jun 21 2021 15:15:34) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.6, Copyright (c) Zend Technologies
    with Yasd v0.3.8, Our Copyright, by codinghuang

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.7.0
Built => Aug 12 2021 12:58:54
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.1.1f  31 Mar 2020
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608
longlang/phpkafka                       v1.1.5     A kafka client. Support php-fpm and Swoole.
[1]-  Done                    php -v
[2]+  Done                    php --ri swoole


  • 提供最小可复现代码:
    public function send(Producer $producer)
    {

        $recordHeader = new RecordHeader();
        $recordHeader->setHeaderKey('test-h');
        $recordHeader->setValue('goods-info');
        $headers = [$recordHeader];
        $producer->send('local_hyperf', 'value', 'key',$headers);
        dump('send...');

    }
/**
 * @Consumer(topic="local_hyperf", nums=1, groupId="hyperf", autoCommit=true)
 */
class DemoKafkaConsumer extends AbstractConsumer
{
    public function consume(ConsumeMessage $message): string
    {
        dump($message->getHeaders());
        dump($message->getValue());
        Log::info(milliseconds() . ' ' . $message->getTopic() . ':' . $message->getKey() . ':' . $message->getValue());
        return Result::ACK;
    }
}

打印$message->getHeaders() 未空

updateBrokers方法bug

*updateBrokers 方法有段代码如下,传入的是主题是空数组

 $response = $this->updateMetadata([], $client);

  • updateMetadata 方法里有获取主题为null 但获取主题 返回类型是array 抛出异常
Return value of longlang\\phpkafka\\Protocol\\Metadata\\MetadataResponse::getTopics() must be of the
type array, null returned

设置了exception callback 可能会死循环

  • 问题代码&修复方案

// SwooleClient.php

   private function startRecvCo(): void
    {
        $this->coRecvRunning = true;
        $this->recvCoId = true;
        $this->recvCoId = Coroutine::create(function () {
            while ($this->coRecvRunning) {
                try {
                    $data = $this->socket->recv(4, -1);
                    if ($data === '') {
                        break;
                    }
                    $length = Int32::unpack($data);
                    $data = $this->socket->recv($length);
                    $correlationId = Int32::unpack($data);
                    if (isset($this->recvChannels[$correlationId])) {
                        $this->recvChannels[$correlationId]->push($data);
                    }
                } catch (Exception $e) {
                    if ($e instanceof SocketException && ! $this->connected) {
                        return;
                    }
                    $callback = $this->getConfig()->getExceptionCallback();
                    if ($callback) {
                        //  已经设置了 exception callback 没有退出就会一直死循环下去
                        $callback($e);
                        // 修复方案:在此次增加 return 
                        return;
                    } else {
                        throw $e;
                    }
                }
            }
        });
    }

消费者过段时间就报错

  • 你遇到了什么问题?
    vendor/longlang/phpkafka/src/Client/SwooleClient.php(135): longlang\phpkafka\Socket\SwooleSocket->recv()
    #1 {main}
    thrown in /opt/code/novel/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php on line 128
    PHP Fatal error: Uncaught longlang\phpkafka\Exception\SocketException: Could not recv data from stream,

  • Kafka 环境是自建还是云服务?

docker自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里

 php -v & php --ri swoole & composer info | grep longlang/phpkafka
[1] 8319
[2] 8320
PHP 7.4.24 (cli) (built: Oct  7 2021 05:44:58) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.7.1
Built => Oct  7 2021 05:50:26
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.1.1  11 Sep 2018
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608
longlang/phpkafka                  v1.2.0    A kafka client. Support php-fpm and Swoole.
[1]-  Done                    php -v
[2]+  Done                    php --ri swoole

* 提供最小可复现代码:

```php
// 你的代码

作为一个生产者,如何确定正确的brokerId?

看了一下代码,需要手动指定brokerId,如果不指定,就随机分配。可是随机分配如果不是对应分区的leader,就会报出错误[6] For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.

是需要用户根据meta、topic和partition自己计算正确的brokerId吗?

Invalid length 352518912 given, it should be lesser than or equals to 5242880

  • 你遇到了什么问题?

prod.ERROR: Invalid length 352518912 given, it should be lesser than or equals to 5242880 {"exception":"[object] (longlang\phpkafka\Exception\SocketException(code: 0): Invalid length 352518912 given, it should be lesser than or equals to 5242880 at /web/vendor/longlang/phpkafka/src/Socket/StreamSocket.php:173)
[stacktrace]

  • Kafka 环境是自建还是云服务?

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里
[1] 7531
[2] 7532
PHP 7.4.30 (cli) (built: Jul 12 2022 09:31:30) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.4.26
Built => Jul 27 2022 03:11:37
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.1.1n  15 Mar 2022
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
Do not run Composer as root/super user! See https://getcomposer.org/root for details
  • 提供最小可复现代码:
// 你的代码
 public function recv(int $length, ?float $timeout = null): string
    {
        if ($length > self::READ_MAX_LENGTH) {
            throw new SocketException(sprintf('Invalid length %d given, it should be lesser than or equals to %d', $length, self::READ_MAX_LENGTH));
        }

        if (null === $timeout) {
            $timeout = $this->config->getRecvTimeout();
        }
        $readable = $this->select([$this->socket], $timeout);

        if (false === $readable) {
            $this->close();
            throw new SocketException(sprintf('Could not read %d bytes from stream (not readable)', $length));
        }

        if (0 === $readable) { // select timeout
            $res = $this->getMetaData();
            $this->close();

            if (!empty($res['timed_out'])) {
                throw new SocketException(sprintf('Timed out reading %d bytes from stream', $length));
            }

            throw new SocketException(sprintf('Could not read %d bytes from stream (not readable)', $length));
        }

        $remainingBytes = $length;
        $data = $chunk = '';

        while ($remainingBytes > 0) {
            $chunk = fread($this->socket, $remainingBytes);

            if (false === $chunk || 0 === \strlen($chunk)) {
                // Zero bytes because of EOF?
                if (feof($this->socket)) {
                    $this->close();
                    throw new SocketException(sprintf('Unexpected EOF while reading %d bytes from stream (no data)', $length));
                }
                // Otherwise wait for bytes
                $readable = $this->select([$this->socket], $timeout);
                if (1 !== $readable) {
                    $this->close();
                    throw new SocketException(sprintf('Timed out while reading %d bytes from stream, %d bytes are still needed', $length, $remainingBytes));
                }

                continue; // attempt another read
            }

            $data .= $chunk;
            $remainingBytes -= \strlen($chunk);
        }

        return $data;
    }

同时启动两个消费者,只有一个能成功拉取到消息。

在YII的console里面启动两个消费者test_work和test_logs分别订阅queue_work和queue_logs。同时向两个主题中发送消息但是只有一个进程能成功收到消息。

Kafka:

public function start() {
         $config = new ConsumerConfig();
         $config->setBroker('127.0.0.1:9092');
         $config->setGroupId('0'); // 分组ID
         $config->setInterval(0.1);
         $config->setTopic('queue_work'); // 主题名称
         $config->setClientId('test_work'); // 客户端ID
         $config->setGroupInstanceId('test_work'); // 分组实例ID

        $consumer = new Consumer( $config, 'consume');
        $consumer->start();
    }
public function logs() {
         $config = new ConsumerConfig();
         $config->setBroker('127.0.0.1:9092');
         $config->setGroupId('0'); // 分组ID
         $config->setInterval(0.1);
         $config->setTopic('queue_logs'); // 主题名称
         $config->setClientId('test_logs'); // 客户端ID
         $config->setGroupInstanceId('test_logs'); // 分组实例ID

        $consumer = new Consumer( $config,'consume');
        $consumer->start();
    }

Action:

public function actionSwoole(){
        try {
            Coroutine::create(function (){
                $kafka = new PKafka();
                $kafka->start();
            });
        }catch (\Exception $e){
            echo $e->getMessage();
        }
    }
public function actionSyn(){
        try {
            Coroutine::create(function (){
                $kafka = new PKafka();
                $kafka->logs();
            });
        }catch (\Exception $e){
            echo $e->getMessage();
        }
    }

在服务端查看结果变成 同个消费者订阅两个主题。
1

能否返回Producer的调用结果

  • 你遇到了什么问题?
    调用Producer的send后,没有返回调用结果。能否支持下,返回调用结果。

  • Kafka 环境是自建还是云服务?
    自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

[1] 31476
[2] 31477
PHP 7.2.34 (cli) (built: Mar 14 2021 18:35:27) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
Extension 'swoole' not present.
[2]  + 31477 exit 1     php --ri swoole
[1]  + 31476 done       php -v
longlang/phpkafka                  v1.2.0    A kafka client. Support php-fpm and Swoole.

  • 提供最小可复现代码:
$config = new ProducerConfig();
$config->setBootstrapServer('127.0.0.1:9092');
$config->setUpdateBrokers(true);
$config->setAcks(-1);
$producer = new Producer($config);
$topic = 'test01';
$value = 'a ';
$producer->send($topic, $value);

Idea: rename bootstrapServer to bootstrapServers and move to common configs

I think it is a common configuration key for both Producers and Consumers to know what are the bootstrapServers and I'd like to enforce the ability to know more than one bootstrapServer using the CSV pattern, it very common on production-grade environments to have more than one broker.
Or the name could be brokerList.

增加auto create topic功能

Kafka已开启kafka_auto_create_topics_enable=true
还是会出现This server does not host this topic-partition.

SwooleSocket报错

Stack trace:
#0 vendor/longlang/phpkafka/src/Client/SwooleClient.php(130): longlang\phpkafka\Socket\SwooleSocket->recv()
#1 {main}
thrown in vendor/longlang/phpkafka/src/Socket/SwooleSocket.php on line 132
[mutex.cc : 2338] RAW: Check w->waitp->cond == nullptr failed: Mutex::Fer while waiting on Condition
PHP Fatal error: Uncaught longlang\phpkafka\Exception\SocketException: Could not recv data from stream, [0] in vendor/longlang/phpkafka/src/Socket/SwooleSocket.php:132

  • Kafka 环境是

自建

Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team [email protected]
Version => 4.6.4
Built => Mar 22 2021 11:44:09
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.0.2k-fips 26 Jan 2017
pcre => enabled
zlib => 1.2.7
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608

# PHP 7.4.12 (cli) (built: Dec 30 2020 11:56:19) ( NTS DEBUG )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies


IDEA 无法自动提示

安装:
composer require longlang/phpkafka

框架:
hyperf ~2.0

问题:
IDEA 无法自动提示

image

只能连本机的9092端口,连接远程会提示Could not connect to tcp://localhost:9092

  • 你遇到了什么问题?

只能连本机的9092端口,连接远程会提示:Could not connect to tcp://localhost:9092 (Connection refused [111]) File:...../vendor/longlang/phpkafka/src/Socket/SwooleSocket.php Line:95
我的ip和端口号都不是localhost和9092,不知道为什么还是会连接到localhost:9092上

  • Kafka 环境是自建还是云服务?

自建,尝试使用华为云的kafka会报一个内存溢出的错误,所以先放弃了,华为云的提示为allowed memory size of 134... bytes exhausted... File:../longlang/phpkafka/src/Broker.php Line136

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里

[1] 22822
[2] 22823
PHP 7.3.27 (cli) (built: Apr 10 2021 14:10:50) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.27, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.27, Copyright (c) 1999-2018, by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.6.5
Built => Apr 20 2021 10:16:05
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
Composer could not find a composer.json file in /usr/src/app/example
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section
[1]-  Done                    php -v
[2]+  Done                    php --ri swoole

* 提供最小可复现代码:

```php
        $config = new ProducerConfig();
        $config->setBootstrapServer(['124.xxx.xxx.xxx,9095']);
        $config->setUpdateBrokers(true);
        $config->setAcks(-1);
        $producer = new Producer($config);
        $topic = 'test333';
        $value = (string) microtime(true);
        $key = uniqid('', true);
        $producer->send($topic, $value, $key);

在socket被断开的时候,可不可以支持broker的socket断线重连

  • 你遇到了什么问题?
    在长时间没生产消息时,socket应该有可能会被断开,有没有什么方法去重连被断开的socket;或者后续可不可以支持在config中配置重连次数,在socket发送或接受消息失败后,去重连指定次数socket

  • Kafka 环境是自建还是云服务?
    自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

PHP 7.4.19 (cli) (built: May 31 2021 04:55:16) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Yasd v0.3.8, Our Copyright, by codinghuang

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.4.15
Built => May 31 2021 05:59:33
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.0.2k-fips  26 Jan 2017
http2 => enabled
pcre => enabled
zlib => 1.2.7
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
Do not run Composer as root/super user! See https://getcomposer.org/root for details
Continue as root/super user [yes]? 
longlang/phpkafka                  v1.1.4  A kafka client. Support php-fpm and Swoole.

  • 提供最小可复现代码:
<?php

use longlang\phpkafka\Producer\Producer;
use longlang\phpkafka\Producer\ProducerConfig;

require './vendor/autoload.php';

$config = new ProducerConfig();
$config->setBootstrapServer('192.168.0.252:9092');
$config->setUpdateBrokers(true);

$config->setAcks(-1);
$producer = new Producer($config);
while (true) {
    $producer->send('mq', date('Y-m-d H:i:s'), uniqid('', true));
    sleep(100);
}

拿这个测试了一下,在发送几条消息后,就抛出socket异常了

异常信息:
PHP Fatal error: Uncaught longlang\phpkafka\Exception\SocketException: Unexpected EOF while reading 4 bytes from stream (no data) in /mount/vagrant/learn/es/vendor/longlang/phpkafka/src/Socket/StreamSocket.php:201
Stack trace:
#0 /mount/vagrant/learn/es/vendor/longlang/phpkafka/src/Client/SyncClient.php(164): longlang\phpkafka\Socket\StreamSocket->recv(4)
#1 /mount/vagrant/learn/es/vendor/longlang/phpkafka/src/Producer/Producer.php(153): longlang\phpkafka\Client\SyncClient->recv(3)
#2 /mount/vagrant/learn/es/vendor/longlang/phpkafka/src/Producer/Producer.php(51): longlang\phpkafka\Producer\Producer->sendBatch(Array, 1)
#3 /mount/vagrant/learn/es/prod.php(20): longlang\phpkafka\Producer\Producer->send('mq', '2021-08-18 02:0...', '611c6bbeb61990....')
#4 {main}
thrown in /mount/vagrant/learn/es/vendor/longlang/phpkafka/src/Socket/StreamSocket.php on line 201

Support for RedPanda?

  • What problem did you encounter?

PHP Fatal error: Uncaught longlang\phpkafka\Exception\KafkaErrorException: [35] The version of API is not supported. in /src/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php:385
Stack trace:
#0 /src/vendor/longlang/phpkafka/src/Util/KafkaUtil.php(69): longlang\phpkafka\Protocol\ErrorCode::check()
#1 /src/vendor/longlang/phpkafka/src/Group/GroupManager.php(111): longlang\phpkafka\Util\KafkaUtil::retry()
#2 /src/vendor/longlang/phpkafka/src/Consumer/Consumer.php(162): longlang\phpkafka\Group\GroupManager->syncGroup()
#3 /src/vendor/longlang/phpkafka/src/Consumer/Consumer.php(128): longlang\phpkafka\Consumer\Consumer->rejoin()
#4 /src/Lib/AbstractServiceEventWorker.php(53): longlang\phpkafka\Consumer\Consumer->__construct()
#5 /src/App/cli/serviceEventWorker.php(19): Lib\AbstractServiceEventWorker->start()

  • Is the Kafka environment self-built or cloud service?

apt installation

  • Please execute the following command to get environment information.

php -v & php --ri swoole & composer info | grep longlang/phpkafka
php8.0 -v & php8.0 --ri openswoole
[1] 533
PHP 8.0.13 (cli) (built: Nov 22 2021 09:50:24) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.13, Copyright (c) Zend Technologies
with Zend OPcache v8.0.13, Copyright (c), by Zend Technologies
with Xdebug v3.0.4, Copyright (c) 2002-2021, by Derick Rethans

openswoole

Open Swoole => enabled
Author => Open Swoole Group & Contributors [email protected]
Version => 4.8.1
Built => Dec 6 2021 17:59:05
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.1.1 11 Sep 2018
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

"longlang/phpkafka": "^1.2",

# Paste here

  • Provide the smallest reproducible code:
// Your code
         $config = new ConsumerConfig();
      $addr = Config::getBroker();
      $config->setBroker($addr->toIPPortFormat());
      $config->setTopic($this->getTopic()); // topic
      $config->setClientId($this->sanitizeClassName(static::class)); // client ID. Use
      $config->setInterval(0.1);
      $config->setSsl(Config::getSSLConfig());
      $config->setGroupId($this->sanitizeClassName($this->getGroupId()));
      $config->setGroupInstanceId($this->sanitizeClassName(static::class));
      $this->config = $config;


......

        $this->consumer = new Consumer($this->config, [$this,'consume']);

        $this->consumer->start();

I am able to connect and consume topics using other php kafka libraries such as simple-kafka-client

 $builder = KafkaConsumerBuilder::create();

        $consumer = $builder->withAdditionalConfig(
            [
                // start at the very beginning of the topic when reading for the first time
                'auto.offset.reset' => 'earliest',

                // will be visible in broker logs
                'client.id' => 'php-kafka-lib-high-level-consumer',

                // SSL settings
                'security.protocol' => 'ssl',
                'ssl.ca.location' => Config::CA_CERT,
                'ssl.certificate.location' => Config::CLIENT_CERT,
                'ssl.key.location' => Config::CLIENT_KEY,

                // SASL settings
                //'sasl.mechanisms' => '',
                //'ssl.endpoint.identification.algorithm' => 'https',
                //'sasl.username' => '',
                //'sasl.password' => '',

                // Add additional output if you need to debug a problem
                // 'log_level' => (string) LOG_DEBUG,
                // 'debug' => 'all'
            ]
        )
            ->withAdditionalBroker(Config::getBroker()->toIPPortFormat())
            ->withConsumerGroup('php-kafka-lib-high-level-consumer')
            ->withSubscription('php-kafka-lib-test-topic')
            ->build();

        $consumer->subscribe();

        while (true) {
            try {
                $message = $consumer->consume(10000);
            } catch (KafkaConsumerTimeoutException|KafkaConsumerEndOfPartitionException $e) {
                echo 'Didn\'t receive any messages, waiting for more...' . PHP_EOL;
                continue;
            } catch (KafkaConsumerConsumeException $e) {
                echo $e->getMessage() . PHP_EOL;
                continue;
            }

            echo sprintf(
                    'Read message with key:%s payload:%s topic:%s partition:%d offset:%d headers:%s',
                    $message->getKey(),
                    $message->getBody(),
                    $message->getTopicName(),
                    $message->getPartition(),
                    $message->getOffset(),
                    implode(',', $message->getHeaders())
                ) . PHP_EOL;

            $consumer->commit($message);
        }

phpkafka的topic不支持正则?

  • 你遇到了什么问题
    topic 含正则表达式 都不能用,显示以下错误, kafka [api]里kafkaConsumer.subscribe 的第一个参数 支持正则模式订阅,
Exception 'longlang\phpkafka\Exception\KafkaErrorException' with message '[17] The request attempted to perform an operation on an invalid topic.'

in /mnt/d/proj_path/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php:385

Stack trace:
#0 /mnt/d/proj_path/vendor/longlang/phpkafka/src/Broker.php(126): longlang\phpkafka\Protocol\ErrorCode::check()
  • Kafka 环境是自建还是云服务?
    自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

[1] 6007
[2] 6008
PHP 7.4.3 (cli) (built: Oct  6 2020 15:47:56) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.3, Copyright (c), by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.6.7
Built => May 25 2021 21:55:02
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.1.1f  31 Mar 2020
dtls => enabled
http2 => enabled
json => enabled
pcre => enabled
zlib => 1.2.11
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608
longlang/phpkafka                  dev-master c7b5082 A kafka client. Support php-fpm and Swoole.
[1]-  Done                    php -v
[2]+  Done                    php --ri swoole
  • 提供最小可复现代码:
php

 $config = new ConsumerConfig($option);
 $config->setBroker(Yii::$app->params['KAFKA']['host']);
 $config->setTopic($this->topic); //这个值 要含正则时 才能触发

一个消费者组下两个消费者,只有一个能消费消息

  • 你遇到了什么问题?

一个消费者组下两个消费者,只有一个能消费消息

  • Kafka 环境是自建还是云服务?

docker自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里
[1] 15367
[2] 15368
Extension 'swoole' not present.
PHP 7.3.0beta1 (cli) (built: Mar 12 2020 12:13:45) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.0-dev, Copyright (c) 1998-2018 Zend Technologies
Do not run Composer as root/super user! See https://getcomposer.org/root for details
longlang/phpkafka          v1.1.5 A kafka client. Support php-fpm and Swoole.
[1]-  完成                  php -v
[2]+  退出 1                php --ri swoole

  • 提供最小可复现代码:
// 你的代码
use longlang\phpkafka\Consumer\ConsumeMessage;
use longlang\phpkafka\Consumer\Consumer;
use longlang\phpkafka\Consumer\ConsumerConfig;
function consume(ConsumeMessage $message)
{
    var_dump($message->getKey() . ':' . $message->getValue());
    // $consumer->ack($message); // autoCommit设为false时,手动提交
}

$rand = rand(1000,9999);

$config = new ConsumerConfig();
$config->setBroker('192.168.1.166:32769');
$config->setTopic('test'); // 主题名称
$config->setGroupId('testGroup'); // 分组ID
$config->setClientId($rand); // 客户端ID,不同的消费者进程请使用不同的设置
$config->setGroupInstanceId($rand); // 分组实例ID,不同的消费者进程请使用不同的设置
$config->setInterval(0.1);
$consumer = new Consumer($config, 'consume');
$consumer->start();

updateApiVersions 报错 [ERROR] [10753] Unknown[385]

  • 你遇到了什么问题?
    报错日志如下,多发于hyperf一段时间无访问时再次访问出错。频繁访问没有报错
    PHP Warning: Uninitialized string offset 0 in /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Protocol/Type/UVarInt.php on line 41

Warning: Uninitialized string offset 0 in /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Protocol/Type/UVarInt.php on line 41
[ERROR] [10753] Unknown[385] in /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php
[ERROR] #0 /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Client/SyncClient.php(196): longlang\phpkafka\Protocol\ErrorCode::check(10753)
#1 /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Client/SyncClient.php(103): longlang\phpkafka\Client\SyncClient->updateApiVersions()
#2 /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Client/SwooleClient.php(52): longlang\phpkafka\Client\SyncClient->connect()
#3 /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Broker.php(174): longlang\phpkafka\Client\SwooleClient->connect()
#4 /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Broker.php(156): longlang\phpkafka\Broker->getClientByBrokerId(1001)
#5 /www/hyperf-skeleton/vendor/longlang/phpkafka/src/Producer/Producer.php(160): longlang\phpkafka\Broker->getClient(1001)
#6 /www/hyperf-skeleton/vendor/hyperf/kafka/src/Producer.php(94): longlang\phpkafka\Producer\Producer->sendBatch(Array)
#7 [internal function]: Hyperf\Kafka\Producer->Hyperf\Kafka{closure}()
#8 /www/hyperf-skeleton/vendor/hyperf/kafka/src/Producer.php(148): Closure->call(Object(Hyperf\Kafka\Producer))
#9 {main}

  • Kafka 环境是自建还是云服务?
    docker环境
    cat docker-compose.yml
version: '2'
services:
  zookeeper:
    image: zookeeper:3.5
    ports:
      - "2182:2181"
  kafka:
    image: wurstmeister/kafka
    depends_on: [ zookeeper ]
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: 192.168.0.90
      KAFKA_CREATE_TOPICS: "cps_static:1:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      #KAFKA_HOME: "/opt/kafka_2.12-2.4.1"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime
  kafka-manager:
    image: kafkamanager/kafka-manager
    depends_on:
      - zookeeper
    ports:
      - "8084:9000"
    environment:
      ZK_HOSTS: zookeeper:2181
  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

[1] 72185
[2] 72186
PHP 8.0.12 (cli) (built: Oct 21 2021 14:38:26) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.12, Copyright (c) Zend Technologies
    with Yasd v0.3.9-alpha, Our Copyright, by codinghuang
[1]  - 72185 done       php -v

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 5.0.0
Built => Aug 23 2022 18:14:26
coroutine => enabled with boost asm context
kqueue => enabled
rwlock => enabled
openssl => OpenSSL 3.0.0 7 sep 2021
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.11
brotli => E16777225/D16777225
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 262144 => 262144
[2]  + 72186 done       php --ri swoole
longlang/phpkafka                       v1.2.1  A kafka client. Support php-fpm and Swoole.

  • 提供最小可复现代码:
无法复现

支持设置producer端参数么

  • 你遇到了什么问题?
    请问支持设置以下kafka的producer端配置么。
    1 retries 就是producer重试的次数设置
    2 max.in.flight.requests.per.connection producer的IO线程在单个Socket连接上能够发送未应答produce请求的最大数量

在代码里面没看到有地方可以设置

在swoole服务中, kafka生产者每过一段时间就报错

  • 你遇到了什么问题?
    在swoole服务中, kafka生产者每过一段时间就报错

  • Kafka 环境是自建还是云服务?
    自建

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

PHP 7.4.25 (cli) (built: Oct 19 2021 15:18:10) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.25, Copyright (c), by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.6.1
Built => Jan 11 2021 12:30:02
coroutine => enabled with boost asm context
trace_log => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
cpu_affinity => enabled
spinlock => enabled
rwlock => enabled
sockets => enabled
openssl => OpenSSL 1.0.2k-fips  26 Jan 2017
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.7
mutex_timedlock => enabled
pthread_barrier => enabled
futex => enabled
mysqlnd => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => On => On
swoole.unixsock_buffer_size => 8388608 => 8388608

  • 提供最小可复现代码:
// 你的代码
declare(strict_types=1);//语法严格模式
require_once dirname(__DIR__).'/vendor/autoload.php';
ini_set('memory_limit', '2048M');
use longlang\phpkafka\Producer\Producer;
use longlang\phpkafka\Producer\ProducerConfig;

$http = new Swoole\Http\Server("0.0.0.0", 9503);
$http->set([
    'tcp_fastopen' => true,
    'max_wait_time' => 1,
    'enable_reuse_port'=>false,//同一个端口复用
    'log_file'=>dirname(__DIR__)."/log/swoole.log"
]);

$config = new ProducerConfig();
$config->setBootstrapServer('172.18.50.74:9092,172.18.50.75:9092');
$config->setUpdateBrokers(true);
$config->setAcks(-1);
$producer = new Producer($config);
$http->on('request', function ($request, $response) use ($producer) {
    try {
        $topic = 'play_history_pre3';
        var_dump($request->server['request_uri']);
        $sendData = json_encode($request->post);
        $producer->send($topic,$sendData);
        unset($sendData,$topic);
        $response->end('success');
    } catch (\Throwable $e) {
        var_dump($e->getMessage());
        $response->end(json_encode(array(
            'code' => '999',
            'message' => '记录失败'
        )));
    }
});

$http->start();
  • 错误信息
PHP Fatal error:  Uncaught longlang\phpkafka\Exception\SocketException: Could not recv data from stream,  [0] in /data/www/kafka/swoole/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php:129
Stack trace:
#0 /data/www/kafka/swoole/vendor/longlang/phpkafka/src/Client/SwooleClient.php(135): longlang\phpkafka\Socket\SwooleSocket->recv(4, -1)
#1 {main}
  thrown in /data/www/kafka/swoole/vendor/longlang/phpkafka/src/Socket/SwooleSocket.php on line 129

我配置bootstrap_servers为内网地址的时候发现链接地址被重置为了 127.0.0.1

  • 你遇到了什么问题?

我配置bootstrap_servers为内网地址的时候发现链接地址被重置为了 127.0.0.1

  • Kafka 环境是自建还是云服务?

docker 环境

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

# 粘贴到这里
PHP 7.4.29 (cli) (built: Apr 23 2022 07:09:26) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.29, Copyright (c), by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.4.24
Built => Jun 15 2022 08:05:05
coroutine => enabled
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.1.1o  3 May 2022
http2 => enabled
pcre => enabled
zlib => 1.2.12
brotli => E16777225/D16777225
mutex_timedlock => enabled
pthread_barrier => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608
longlang/phpkafka                   v1.2.1             A kafka client. Support php-fpm and Swoole.

  • 提供最小可复现代码:
    7R0RK0ST2`6U2ULSCR4{$ZR
    X%50M$TYT GV9_Y$20 LR73
    75$YO}$J24U$48VCKX O1VB

[16] This is not the correct coordinator

docker运行
PHP 7.2.29 , swoole 4.5.7
kafka 官方镜像: wurstmeister/kafka:2.12-2.5.0

使用代码为官方demo, produce成功,consumer失败。
Fatal error: Uncaught longlang\phpkafka\Exception\KafkaErrorException: [16] This is not the correct coordinator. in /var/www/html/vendor/longlang/phpkafka/src/Protocol/
ErrorCode.php:385
Stack trace:
#0 /var/www/html/vendor/longlang/phpkafka/src/Consumer/OffsetManager.php(78): longlang\phpkafka\Protocol\ErrorCode::check(16)
#1 /var/www/html/vendor/longlang/phpkafka/src/Consumer/Consumer.php(70): longlang\phpkafka\Consumer\OffsetManager->updateOffsets()
#2 /var/www/html/test/kafka/consumer.php(14): longlang\phpkafka\Consumer\Consumer->__construct(Object(longlang\phpkafka\Consumer\ConsumerConfig))
#3 {main}
thrown in /var/www/html/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php on line 385

设置group信息后同样报错

「Suggestion」brokers 设置为 null

看到支持 bootstrap_server,所以,想着能否将 brokers 直接设置为 null

于是看了代码,发现代码中是支持的:https://github.com/longyan/phpkafka/blob/master/src/Broker.php#L71

因此,麻烦看看,能否

  1. https://github.com/longyan/phpkafka/blob/master/src/Consumer/ConsumerConfig.php#L26
  2. https://github.com/longyan/phpkafka/blob/master/src/Producer/ProducerConfig.php#L27
  3. 文档

的注释或文档中,加上 null 同时说明下咧

PS: 实际测试也是 ok 的

非常感谢

For requests intended only for the leader

PHP Fatal error: Uncaught longlang\phpkafka\Exception\KafkaErrorException: [6] For requests intended only for the leader, this error indicates that the broker is not the current leader. For requests intended for any replica, this error indicates that the broker is not a replica of the topic partition.

Fatal error: Uncaught longlang\phpkafka\Exception\KafkaErrorException

  • 你遇到了什么问题?
PHP Warning:  unpack(): Type l: not enough input, need 4, have 0 in /www/vendor/longlang/phpkafka/src/Protocol/Type/Int32.php on line 43

Warning: unpack(): Type l: not enough input, need 4, have 0 in /www/vendor/longlang/phpkafka/src/Protocol/Type/Int32.php on line 43
PHP Fatal error:  Uncaught longlang\phpkafka\Exception\KafkaErrorException: [35] The version of API is not supported. in /www/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php:385
Stack trace:
#0 /www/vendor/longlang/phpkafka/src/Client/SyncClient.php(190): longlang\phpkafka\Protocol\ErrorCode::check()
#1 /www/vendor/longlang/phpkafka/src/Client/SyncClient.php(98): longlang\phpkafka\Client\SyncClient->updateApiVersions()
#2 /www/vendor/longlang/phpkafka/src/Client/SwooleClient.php(52): longlang\phpkafka\Client\SyncClient->connect()
#3 /www/vendor/longlang/phpkafka/src/Broker.php(88): longlang\phpkafka\Client\SwooleClient->connect()
#4 /www/vendor/longlang/phpkafka/src/Producer/Producer.php(39): longlang\phpkafka\Broker->updateBrokers()
#5 /www/vendor/hyperf/kafka/src/Producer.php(173): longlang\phpkafka\Producer\Producer->__construct()
#6 /www/vendor/hyperf/kafka/src/Producer.php(137): Hyperf\Kafka\Producer->makeProducer()
#7 {main}
  thrown in /www/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php on line 385

Fatal error: Uncaught longlang\phpkafka\Exception\KafkaErrorException: [35] The version of API is not supported. in /www/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php:385
Stack trace:
#0 /www/vendor/longlang/phpkafka/src/Client/SyncClient.php(190): longlang\phpkafka\Protocol\ErrorCode::check()
#1 /www/vendor/longlang/phpkafka/src/Client/SyncClient.php(98): longlang\phpkafka\Client\SyncClient->updateApiVersions()
#2 /www/vendor/longlang/phpkafka/src/Client/SwooleClient.php(52): longlang\phpkafka\Client\SyncClient->connect()
#3 /www/vendor/longlang/phpkafka/src/Broker.php(88): longlang\phpkafka\Client\SwooleClient->connect()
#4 /www/vendor/longlang/phpkafka/src/Producer/Producer.php(39): longlang\phpkafka\Broker->updateBrokers()
#5 /www/vendor/hyperf/kafka/src/Producer.php(173): longlang\phpkafka\Producer\Producer->__construct()
#6 /www/vendor/hyperf/kafka/src/Producer.php(137): Hyperf\Kafka\Producer->makeProducer()
#7 {main}
  thrown in /www/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php on line 385
[2021-09-02 02:58:02 *753.1]	ERROR	php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Uncaught longlang\phpkafka\Exception\KafkaErrorException: [35] The version of API is not supported. in /www/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php:385
Stack trace:
#0 /www/vendor/longlang/phpkafka/src/Client/SyncClient.php(190): longlang\phpkafka\Protocol\ErrorCode::check()
#1 /www/vendor/longlang/phpkafka/src/Client/SyncClient.php(98): longlang\phpkafka\Client\SyncClient->updateApiVersions()
#2 /www/vendor/longlang/phpkafka/src/Client/SwooleClient.php(52): longlang\phpkafka\Client\SyncClient->connect()
#3 /www/vendor/longlang/phpkafka/src/Broker.php(88): longlang\phpkafka\Client\SwooleClient->connect()
#4 /www/vendor/longlang/phpkafka/src/Producer/Producer.php(39): longlang\phpkafka\Broker->updateBrokers()
#5 /www/vendor/hyperf/kafka/src/Producer.php(173): longlang\phpkafka\Producer\Producer->__construct()
#6 /www/vendor/hyperf/kafka/src/Producer.php(137): Hyperf\Kafka\Producer->makeProducer()
#7 {main}
  thrown in /www/vendor/longlang/phpkafka/src/Protocol/ErrorCode.php on line 385

其中,“The version of API is not supported.”指明phpkafka不支持,不晓得需要什么具体的版本?

  • Kafka 环境是自建还是云服务?
    根据Hypef-2.1手册,使用Docker的hyperf/hyperf:7.4-alpine-v3.13-swoole 镜像建立的容器

  • 请执行下面的命令获取环境信息。

php -v & php --ri swoole & composer info | grep longlang/phpkafka

bash-5.1# php -v & php --ri swoole & composer info | grep longlang/phpkafka
[1] 777
[2] 778
PHP 7.4.23 (cli) (built: Aug 26 2021 23:05:01) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.21, Copyright (c), by Zend Technologies

swoole

Swoole => enabled
Author => Swoole Team <[email protected]>
Version => 4.7.0
Built => Aug 28 2021 09:20:30
coroutine => enabled with boost asm context
epoll => enabled
eventfd => enabled
signalfd => enabled
spinlock => enabled
rwlock => enabled
openssl => OpenSSL 1.1.1l  24 Aug 2021
dtls => enabled
http2 => enabled
json => enabled
curl-native => enabled
pcre => enabled
zlib => 1.2.11
brotli => E16777225/D16777225
mutex_timedlock => enabled
pthread_barrier => enabled
async_redis => enabled

Directive => Local Value => Master Value
swoole.enable_coroutine => On => On
swoole.enable_library => On => On
swoole.enable_preemptive_scheduler => Off => Off
swoole.display_errors => On => On
swoole.use_shortname => Off => Off
swoole.unixsock_buffer_size => 8388608 => 8388608
longlang/phpkafka                    v1.1.5  A kafka client. Support php-fpm and Swoole.
[1]-  Done                    php -v
[2]+  Done                    php --ri swoole

  • 提供最小可复现代码:

kafka配置

<?php

declare(strict_types=1);

use Hyperf\Kafka\Constants\KafkaStrategy;

return [
    'default'     => [
        'connect_timeout'               => -1,
        'send_timeout'                  => -1,
        'recv_timeout'                  => -1,
        'client_id'                     => '',
        'max_write_attempts'            => 3,
        'bootstrap_servers'             => explode(',', env('KAFKA_LOG_BOOTSTRAP_SERVERS', '')),
        'acks'                          => -1,
        'producer_id'                   => -1,
        'producer_epoch'                => -1,
        'partition_leader_epoch'        => -1,
        'interval'                      => 0,
        'session_timeout'               => 60,
        'rebalance_timeout'             => 60,
        'replica_id'                    => -1,
        'rack_id'                       => '',
        'group_retry'                   => 5,
        'group_retry_sleep'             => 1,
        'group_heartbeat'               => 3,
        'offset_retry'                  => 5,
        'auto_create_topic'             => true,
        'partition_assignment_strategy' => KafkaStrategy::RANGE_ASSIGNOR,
        'pool'                          => [
            'min_connections' => 1,
            'max_connections' => 10,
            'connect_timeout' => 10.0,
            'wait_timeout'    => 3.0,
            'heartbeat'       => -1,
            'max_idle_time'   => 60.0,
        ],
    ],
    'log_service' => [
        'app'       => env('KAFKA_LOG_APP', ''),
        'topic'     => env('KAFKA_LOG_TOPIC', '--topic--’),
        'bootstrap' => env('KAFKA_LOG_BOOTSTRAP_SERVERS', '--bootstrap--‘),
    ],
];

<?php
declare(strict_types=1);

namespace App\Service\QueueService;

use App\Service\PageService\BasePS;
use Hyperf\Contract\ConfigInterface;
use Hyperf\HttpServer\Contract\RequestInterface;
use Hyperf\Kafka\Producer;
use Hyperf\Logger\LoggerFactory;
use Hyperf\Contract\ContainerInterface;

class KafkaQS
{

    protected $producer;
    protected $logger;
    protected $config;
    protected $container;
    protected $request;

    public function __construct(ContainerInterface $container, Producer $producer, RequestInterface $request)
    {
        $this->producer  = $producer;
        $this->container = $container;
        $this->config    = $container->get(ConfigInterface::class)->get('kafka.log_service');
    }

    /**
     * elkLog
     *
     * @param array $param
     * @return bool
     */
    public function elkLog(array $param = []): bool
    {
        $topic       = $this->config['topic'];
        $url         = $this->config['bootstrap'];

        $logStr = 'test demo';
        $this->producer->send($topic, $logStr, $url);
        return true;
    }

}

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.