redpanda-data / redpanda Goto Github PK

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!

Emacs Lisp 0.01% Shell 0.10% CMake 0.57% Dockerfile 0.04% Python 19.87% Go 8.65% C++ 68.49% C 0.05% Makefile 0.01% Java 0.71% Jinja 0.01% Rust 1.51% JavaScript 0.01%

kafka cpp kubernetes go streaming redpanda containers realtime event-driven microservices

redpanda's Introduction

Redpanda

Redpanda is a streaming data platform for developers. Kafka® API-compatible. ZooKeeper® free. JVM free. We built it from the ground up to eliminate complexity common to Apache Kafka, improve performance by up to 10x, and make the storage architecture safer, and more resilient. The simpler devex lets you focus on your code (instead of fighting Kafka) and develop new use cases that were never before possible. The business benefits from a significantly lower total cost and faster time to market. A new platform that scales with you from the smallest projects to petabytes of data distributed across the globe!

Community

Slack is the main way the community interacts with one another in real-time :)

Github Discussion is preferred for longer, async, thoughtful discussions

GitHub Issues is reserved only for actual issues. Please use the mailing list for discussions.

Code of conduct code of conduct for the community

Contributing docs

Getting Started

Prebuilt Packages

We recommend using our free & prebuilt stable releases below.

On MacOS

Simply download our rpk binary here. We require Docker on MacOS

brew install redpanda-data/tap/redpanda && rpk container start

On Debian/Ubuntu

curl -1sLf \
  'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' \
  | sudo -E bash

sudo apt-get install redpanda

On Fedora/RedHat/Amazon Linux

curl -1sLf \
  'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' \
  | sudo -E bash

sudo yum install redpanda

On Other Linux

To install from a .tar.gz archive, download the file and extract it into /opt/redpanda.

For amd64:

curl -LO \
  https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/raw/names/redpanda-amd64/versions/23.3.6/redpanda-23.3.6-amd64.tar.gz

For arm64:

curl -LO \
  https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/raw/names/redpanda-arm64/versions/23.3.6/redpanda-23.3.6-arm64.tar.gz

Replace 23.3.6 with the appropriate version you are trying to download.

GitHub Actions

    - name: start redpanda
      uses: redpanda-data/[email protected]
      with:
        version: "latest"

Now you should be able to connect to redpanda (kafka-api) running at localhost:9092

Build Manually

We provide a very simple build system that uses your system libraries. We recommend users leverage our pre-built stable releases which are vetted, tested, and reproducible with exact versions of the entire transitive dependency graph, including exact compilers all built from source. The only thing we do not build yet is the Linux Kernel, but soon!

Currently clang 16 is required. We test the open-source build nightly using Fedora 38.

sudo ./install-dependencies.sh
cmake --preset release
cmake --build --preset release

For quicker dev setup, we provide a docker image with the toolchain installed.

Release candidate builds

We create a release candidate (RC) build when we get close to a new release and publish these to make new features available for testing. RC builds are not recommended for production use.

RC releases on Debian/Ubuntu

curl -1sLf \
  'https://dl.redpanda.com/E4xN1tVe3Xy60GTx/redpanda-unstable/setup.deb.sh' \
  | sudo -E bash

sudo apt-get install redpanda

RC releases on Fedora/RedHat/Amazon Linux

curl -1sLf \
  'https://dl.redpanda.com/E4xN1tVe3Xy60GTx/redpanda-unstable/setup.rpm.sh' \
  | sudo -E bash

sudo yum install redpanda

RC releases on Docker

This is an example with the v23.1.1-rc1 version prior to the 23.1.1 release.

docker pull docker.redpanda.com/redpandadata/redpanda-unstable:v23.1.1-rc1

redpanda's People

Contributors

Stargazers

Watchers

Forkers

emaxerrno dotnwat migue mmaslankaprv graphcareful yshalabi crackcomm 0x5d benpope shravanheroor sthagen cxz doytsujin lazin andresaristizabal rheehot dendisuhubdy kokizzu msurendra 0xdiba rystsov lwelch45 polymath-is cloudmarc nawalgupta jacobjohansen xeron56 zeta1999 rkruze goswamig shaunstanislauslau linecode krambn chsjiang feiyunwill sangrho chenchengshao fernandomora wiltonlazary c4milo ivotron runsafesecurity trucnguyenlam aenglisc fernandadguez rafalkorepta alenkacz triump2020 dimitriscruz nikileshsa sjanulonoks imaginary-person ezhangle mateoc chrislusf hopkings2008 org-mars rogervaas kepler62f beyonddream-productions 96radhikajadhav neuroradiology deepskyblue86 amishsheikh yougotashovel jamesliudotcc fjardon simon0191 shridharr050 markpapadakis tanwanirahul felixguendling rohan-puri lumontec ketoo xiaojian-hong arekzarowski anandspersonal rohankumar997 vadimplh ekingg parisholley perbu canthackthehack aanogueira edric-shen eformat sisco0 nick-zh nitin-mane binglu0158 desertmonkey jasonvaccaro riccardotonini github-hugo-maitre hmflashy jackyng88 tedkirkpatrick skunkwerks easel

redpanda's Issues

Building on PopOS (an ubuntu for System76)

sudo ./install-dependencies.sh fails on PopOS, which is an ubuntu for System76 machines. This is because it's an ubuntu where /etc/os-release looks like this, where ID=pop, so it's no in the list ubuntu debian:

NAME="Pop!_OS"
VERSION="20.04 LTS"
ID=pop
ID_LIKE="ubuntu debian"
PRETTY_NAME="Pop!_OS 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://pop.system76.com"
SUPPORT_URL="https://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
LOGO=distributor-logo-pop-os

I tried to just edit the sudo ./install-dependencies.sh script to just do ubuntu | debian | pop but it still fails later on in the build when it doesn't recognize ID=pop. I think there are multiple places where it deals with equivalent ubuntu-like systems where pop would need to be in the list.

So, I did something stupid and edited /etc/os-release to be ID=ubuntu for the duration of sudo ./install-dependencies.sh, and it seemed to have worked without complaint. I have no idea if I just messed up my system; but it looks ok right now.

I left it that way, and run ./build.sh. It builds, and build looked successful, but some tests failed. As far as I can tell, it's exactly like Ubuntu.

/bin/bash: line 1: 146777 Aborted                 (core dumped) /home/rfielding/redpanda/redpanda/build/bin/pandaproxy_fixture_rpunit -- --overprovisioned --unsafe-bypass-fsync 1 --default-log-level=trace --logger-log-level='io=debug' --logger-log-level='exception=debug' --blocked-reactor-notify-ms 2000000

      Start 82: test_http_client_rpunit
77/77 Test #82: test_http_client_rpunit .....................   Passed    0.35 sec

92% tests passed, 6 tests failed out of 77

Label Time Summary:
compression    =   0.67 sec*proc (1 test)
hashing        =   0.06 sec*proc (1 test)
model          =   0.38 sec*proc (1 test)
pandaproxy     =   3.45 sec*proc (5 tests)
raft           =  32.01 sec*proc (3 tests)
rpc            =   4.73 sec*proc (6 tests)
storage        =  28.23 sec*proc (20 tests)

Total Test time (real) =  96.70 sec

The following tests FAILED:
	 37 - read_materialized_topic_unit_tests_rpunit (Failed)
	 68 - test_cluster_rpunit (Failed)
	 73 - test_kafka_rpunit (Failed)
	 74 - test_kafka_request_parser_rpunit (Failed)
	 78 - pandaproxy_client_fixture_rpunit (Failed)
	 81 - pandaproxy_fixture_rpunit (Failed)
Errors while running CTest

refreshBrokerMetadata causing errors on node-red kafka nodes

When using node-red's currently maintained kafka nodes, refreshBrokerMetadata seems to be causing an error.

This can be reproduced by injecting with either this library and flow:
node-red-contrib-kafka-manager

    {
        "id": "9f287b73.3e8938",
        "type": "Kafka Producer",
        "z": "339485c1.f72bba",
        "name": "Redpanda",
        "broker": "a9e65795.091218",
        "topic": "test_the_topic",
        "topicSlash2dot": false,
        "requireAcks": "1",
        "ackTimeoutMs": 100,
        "partitionerType": 0,
        "key": "0",
        "partition": 0,
        "attributes": 0,
        "connectionType": "Producer",
        "convertFromJson": false,
        "x": 630,
        "y": 280,
        "wires": []
    },
    {
        "id": "62aaa36.0964f5c",
        "type": "inject",
        "z": "339485c1.f72bba",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "test_topic",
        "payload": "{\"payload_test\":\"some value\"}",
        "payloadType": "json",
        "x": 340,
        "y": 280,
        "wires": [
            [
                "9f287b73.3e8938"
            ]
        ]
    },
    {
        "id": "e70ef813.d31ef8",
        "type": "Kafka Consumer",
        "z": "339485c1.f72bba",
        "name": "Redpanda Consumer",
        "broker": "a9e65795.091218",
        "regex": false,
        "topics": [
            {
                "topic": "test_topic",
                "offset": 0,
                "partition": 0
            }
        ],
        "groupId": "kafka-node-group",
        "autoCommit": "true",
        "autoCommitIntervalMs": 5000,
        "fetchMaxWaitMs": 100,
        "fetchMinBytes": 1,
        "fetchMaxBytes": 1048576,
        "fromOffset": 0,
        "encoding": "utf8",
        "keyEncoding": "utf8",
        "connectionType": "Consumer",
        "convertToJson": false,
        "x": 340,
        "y": 420,
        "wires": [
            [
                "4e2684da.f9fcec"
            ]
        ]
    },
    {
        "id": "4e2684da.f9fcec",
        "type": "debug",
        "z": "339485c1.f72bba",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 720,
        "y": 420,
        "wires": []
    },
    {
        "id": "a9e65795.091218",
        "type": "Kafka Broker",
        "name": "Redpanda",
        "hosts": [
            {
                "host": "127.0.0.1",
                "port": 9092
            }
        ],
        "hostsEnvVar": "",
        "connectTimeout": "10000",
        "requestTimeout": "30000",
        "autoConnect": "true",
        "idleConnection": "5",
        "reconnectOnIdle": "true",
        "maxAsyncRequests": "10",
        "checkInterval": "10",
        "tls": "",
        "selfSign": true,
        "usetls": false,
        "useCredentials": false
    }
]`

Or this library and flow:
[Kafka Node](https://flows.nodered.org/node/node-red-contrib-kafka-node)
`[
    {
        "id": "8c38ddd4.853f8",
        "type": "kafka-producer",
        "z": "339485c1.f72bba",
        "name": "Redpanda v2",
        "broker": "74cc8699.1f8b78",
        "topic": "test_topic_2",
        "requireAcks": 1,
        "ackTimeoutMs": 100,
        "attributes": 0,
        "x": 890,
        "y": 600,
        "wires": []
    },
    {
        "id": "b768e15d.d869b",
        "type": "inject",
        "z": "339485c1.f72bba",
        "name": "",
        "props": [
            {
                "p": "payload"
            },
            {
                "p": "topic",
                "vt": "str"
            }
        ],
        "repeat": "",
        "crontab": "",
        "once": false,
        "onceDelay": 0.1,
        "topic": "test_topic_2",
        "payload": "{\"payload_test\":\"some value\"}",
        "payloadType": "json",
        "x": 410,
        "y": 600,
        "wires": [
            [
                "8c38ddd4.853f8"
            ]
        ]
    },
    {
        "id": "bb9f9bea.177da8",
        "type": "kafka-consumer",
        "z": "339485c1.f72bba",
        "name": "Redpanda v2 Consumer",
        "broker": "74cc8699.1f8b78",
        "outOfRangeOffset": "earliest",
        "fromOffset": "latest",
        "topic": "test_topic_2",
        "groupid": "0",
        "x": 460,
        "y": 720,
        "wires": [
            [
                "d9ce00.655ae2"
            ]
        ]
    },
    {
        "id": "d9ce00.655ae2",
        "type": "debug",
        "z": "339485c1.f72bba",
        "name": "",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "false",
        "statusVal": "",
        "statusType": "auto",
        "x": 850,
        "y": 720,
        "wires": []
    },
    {
        "id": "74cc8699.1f8b78",
        "type": "kafka-broker",
        "name": "Redpanda v2",
        "hosts": "localhost:9092",
        "selfsign": false,
        "usetls": false,
        "cacert": "",
        "clientcert": "",
        "privatekey": "",
        "passphrase": ""
    }
]

kafka: partition id is not always set in fetch response error scenarios

Set the default value for target_quota_byte_rate to be max int

Currently today target_quota_byte_rate is set to 64MB/s per client id. It seems this want meant to mimic quota.consumer.default, which has been deprecated in favor of dynamic quotas. So when someone compares Kafka to Redpanda, it is possible they run into this 64MB/s limit with Redpanda but don't see the same behavior in Kafka.

fetch debouncing may produce duplicate entries in response

introduced in #55. description of issue is here #55 (review). simple solution is to reset entire response before retrying. finer grained retry is also possible but will take a bit more accounting and could be handled in a separate optimization ticket.

Problems running a producer / consumer setup using cppkafka

Hi Red Panda developers,

Thanks very much for your amazing contribution to FOSS. Following on from the HN announcement [1], I am trying to play around with Red Panda. First, I apologise in advance as I am not an experienced Kafka user so I am certain I am doing something wrong, but not quite sure what. I am using Debian Testing and I've installed your DEB packages as well as the distro-supplied librdkafka [2] (version 1.51 on Debian). I then checked out and compiled cppkafka [3]. The Red Panda logs appear OK to the untrained eye:

Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,362 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:614 - Recovered, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808, last_term_start_offset:-9223372036854775808}, term:0
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,373 [shard 0] storage - segment.cc:594 - Creating new segment /var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,393 [shard 0] cluster - state_machine.cc:20 - Starting state machine
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,393 [shard 0] cluster - members_manager.cc:46 - starting cluster::members_manager...
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,405 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:225 - became the leader term:1
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,405 [shard 0] storage - segment.cc:594 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,406 [shard 0] redpanda::main - application.cc:468 - Started RPC server listening at {host: 0.0.0.0, port: 33145}
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,406 [shard 0] redpanda::main - application.cc:506 - Started Kafka API server listening at {host: 0.0.0.0, port: 9092}
Nov 16 18:12:56 lovelace rpk[497754]: INFO  2020-11-16 18:12:56,406 [shard 0] redpanda::main - application.cc:508 - Successfully started Redpanda!
Nov 16 18:12:56 lovelace systemd[1]: Started Redpanda, the fastest queue in the West..
Nov 16 18:37:03 lovelace rpk[497754]: INFO  2020-11-16 18:37:03,765 [shard 1] raft - [group_id:1, {kafka_internal/group/0}] consensus.cc:586 - Starting
Nov 16 18:37:03 lovelace rpk[497754]: INFO  2020-11-16 18:37:03,765 [shard 1] raft - [group_id:1, {kafka_internal/group/0}] consensus.cc:614 - Recovered, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808, last_term_start_offset:-9223372036854775808}, term:0
Nov 16 18:37:03 lovelace rpk[497754]: INFO  2020-11-16 18:37:03,775 [shard 1] storage - segment.cc:594 - Creating new segment /var/lib/redpanda/data/redpanda/kvstore/1_0/0-0-v1.log
Nov 16 18:37:03 lovelace rpk[497754]: INFO  2020-11-16 18:37:03,811 [shard 1] raft - [group_id:1, {kafka_internal/group/0}] vote_stm.cc:225 - became the leader term:1
Nov 16 18:37:03 lovelace rpk[497754]: INFO  2020-11-16 18:37:03,811 [shard 1] storage - segment.cc:594 - Creating new segment /var/lib/redpanda/data/kafka_internal/group/0_2/0-1-v1.log

I then started the producer. It errored first but then succeeded:

$ ./examples/producer -b localhost -t test
Producing messages into topic test
%3|1605551753.243|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1605551753.243|ERROR|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
interrupt
$ ./examples/producer -b localhost -t test
Producing messages into topic test

However, when I try to start the consumer I get an error:

$ ./consumer -b localhost -t test -g 0
Consuming messages from topic test
%3|1605565901.190|FAIL|rdkafka#consumer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1605565901.190|ERROR|rdkafka#consumer-1| [thrd:localhost:9092/bootstrap]: 1/1 brokers are down
%3|1605565901.190|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: localhost:9092/bootstrap: Connect to ipv6#[::1]:9092 failed: Connection refused (after 0ms in state CONNECT)
%3|1605565902.191|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 1ms in state CONNECT)
%3|1605565902.191|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 1ms in state CONNECT)
%3|1605565903.191|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1605565903.191|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 0ms in state CONNECT, 1 identical error(s) suppressed)

I tried a few variations such as using 127.0.0.1 instead of localhost because I thought it could be an IPv6 issue, but to no avail:

$ ./consumer -b 127.0.0.1 -t test -g 0
Consuming messages from topic test
%3|1605565953.343|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 0ms in state CONNECT)
%3|1605565953.343|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 0ms in state CONNECT)
%3|1605565954.343|FAIL|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 0ms in state CONNECT, 1 identical error(s) suppressed)
%3|1605565954.343|ERROR|rdkafka#consumer-1| [thrd:app]: rdkafka#consumer-1: GroupCoordinator: Failed to resolve ':0': Name or service not known (after 0ms in state CONNECT, 1 identical error(s) suppressed)

I also tried variations of groups such as 1, etc. I'll keep playing with it, but any pointers as to what I am doing wrong would be greatly appreciated.

Cheers

Marco

[1] https://news.ycombinator.com/item?id=25075739
[2] https://github.com/edenhill/librdkafka
[3] https://github.com/mfontanini/cppkafka

Automatic load balancing of partitions

Redpanda should automatically balance partitions (replicas) and leaders across the cluster's nodes to be as balanced as possible. The balancing should also be done across cores in a single box as well.

It should also be possible to turn off the balancer via a configuration option.

Add simple validation to Kafka request header parsing

We have to add assertion to Kafka requests header parser to prevent seastar assertion from happening:

[Backtrace #0]
 0x0000000001426b1e: void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/include/seastar/util/backtrace.hh:59
 (inlined by) seastar::backtrace_buffer::append_backtrace() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:731
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:752
 (inlined by) seastar::print_with_backtrace(char const*) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:759
 (inlined by) seastar::sigabrt_action() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3486
 (inlined by) operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3468
 (inlined by) __invoke at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3464
 0x0000295a7a5eca8f: ?? ??:0
{/opt/redpanda/lib/libc.so.6} 0x000000000003c9e4: /opt/redpanda/lib/libc.so.6 0x000000000003c9e4 
{/opt/redpanda/lib/libc.so.6} 0x0000000000025894: /opt/redpanda/lib/libc.so.6 0x0000000000025894 
 0x000000000133b977: seastar::memory::abort_on_underflow(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1203
 (inlined by) seastar::memory::allocate_large(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1208
 (inlined by) seastar::memory::allocate(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1269
 0x00000000013471e5: malloc at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1608
 0x0000000000be6166: temporary_buffer at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/temporary_buffer.hh:73
 (inlined by) seastar::input_stream<char>::read_exactly(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/iostream-impl.hh:190
 0x0000000000be465a: operator() at /home/mmaslanka/dev/v/build/release/clang/../../../src/v/kafka/protocol_utils.cc:33
 0x0000000000be2d0c: seastar::future<std::__1::optional<kafka::request_header> > seastar::futurize<seastar::future<std::__1::optional<kafka::request_header> > >::invoke<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::temporary_buffer<char> >(kafka::parse_header(seastar::input_stream<char>&)::$_0&&, seastar::temporary_buffer<char>&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<std::__1::optional<kafka::request_header> > seastar::future<seastar::temporary_buffer<char> >::then_impl<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::future<std::__1::optional<kafka::request_header> > >(kafka::parse_header(seastar::input_stream<char>&)::$_0&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::temporary_buffer<char> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::temporary_buffer<char> > >::run<kafka::parse_header(seastar::input_stream<char>&)::$_0>(seastar::future<seastar::temporary_buffer<char> >&, kafka::parse_header(seastar::input_stream<char>&)::$_0&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<std::__1::optional<kafka::request_header> > seastar::future<seastar::temporary_buffer<char> >::then<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::future<std::__1::optional<kafka::request_header> > >(kafka::parse_header(seastar::input_stream<char>&)::$_0&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1520
 (inlined by) kafka::parse_header(seastar::input_stream<char>&) at /home/mmaslanka/dev/v/build/release/clang/../../../src/v/kafka/protocol_utils.cc:9
 0x0000000000abefde: operator() at /home/mmaslanka/dev/v/build/release/clang/../../../src/v/kafka/protocol.cc:60
 (inlined by) _ZNSt3__18__invokeIRZN5kafka8protocol18connection_context19process_one_requestEvE3$_4JNS_8optionalImEEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS8_DpOS9_ at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/type_traits:3539
 (inlined by) std::__1::invoke_result<kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long> >::type std::__1::invoke<kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long> >(kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long>&&) at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/functional:2902
 (inlined by) auto seastar::internal::future_invoke<kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long> >(kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long>&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1211
 (inlined by) operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1582
 (inlined by) void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<std::__1::optional<unsigned long> >::then_impl_nrvo<kafka::protocol::connection_context::process_one_request()::$_4, seastar::future<void> >(kafka::protocol::connection_context::process_one_request()::$_4&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&, seastar::future_state<std::__1::optional<unsigned long> >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&, seastar::future_state<std::__1::optional<unsigned long> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:2120
 (inlined by) operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1575
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, kafka::protocol::connection_context::process_one_request()::$_4, seastar::future<std::__1::optional<unsigned long> >::then_impl_nrvo<kafka::protocol::connection_context::process_one_request()::$_4, seastar::future<void> >(kafka::protocol::connection_context::process_one_request()::$_4&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&, seastar::future_state<std::__1::optional<unsigned long> >&&)#1}, std::__1::optional<unsigned long> >::run_and_dispose() at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:767
 0x00000000013cd0ef: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2198
 (inlined by) seastar::reactor::run_some_tasks() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2577
addr2line: '/opt/redpanda/lib/libpthread.so.0': No such file
 0x00000000013cfce9: seastar::reactor::run() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2732
 0x000000000142bbe1: operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3908
 0x00000000013642af: std::__1::__function::__value_func<void ()>::operator()() const at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/functional:1867
 (inlined by) std::__1::function<void ()>::operator()() const at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/functional:2473
 (inlined by) seastar::posix_thread::start_routine(void*) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:60
{/opt/redpanda/lib/libpthread.so.0} 0x0000000000009431: /opt/redpanda/lib/libpthread.so.0 0x0000000000009431 
{/opt/redpanda/lib/libc.so.6} 0x0000000000101912: /opt/redpanda/lib/libc.so.6 0x0000000000101912

add readme to docker hub page

#97 (comment)

Error while connecting with Java Kafka driver

I have an application that works pretty well with Kafka, using Akka Streams.
Latest version of Alpakka, brings in dependency on Kafka client 2.4.1

When I point it to Redpanda, this is what I get back:

[2020-11-21 09:04:50,000] [WARN] 
[org.apache.kafka.clients.producer.internals.Sender] 
[kafka-producer-network-thread | producer-1] [] - 
[Producer clientId=producer-1] 
Got error produce response with correlation id 6 on topic-partition postrawdata-0, retrying (2147483646 attempts left). Error: NOT_LEADER_FOR_PARTITION

Is there a configuration option that I'm missing ? What am I doing wrong ?
Thank you!

vectorized ci raft test

Step #6 - "test redpanda": 	 65 - test_raft_rpunit (Failed)```

rpk start: Allow passing --seeds and --advertised

This will make it easier to run in general, but specifically in containers.

support reading past non-committed segment range

So-called dirty reads let us read past the last flushed position of a segment. This is an important optimization for raft recovery workload to avoid an expensive flush operation in order to read recently written data to send to followers.

Effectively the solution tracks a last stable offset which is the highest written (but not necessary flushed) batch offset. The log reader then allows reads up to this stable point from disk, and servers dirty reads (that may be past the stable point but still sitting in a linearization / appender chunk buffer) from the batch cache.

Incremental fetch support in Redpanda

Incremental fetch support was introduced in FetchRequest version 7. We have to add fetch session management to redpanda.

More details:
Kip-227 Incremental Fetch Requests

storage: bound age of unflushed append data

We want to make sure that data in the segment appender write back buffers is written to disk and made visible in a timely manner. Normally this happens as buffers fill up and writes are dispatched, but data may sit in the buffers if the workload idles and does not drive this process.

To address this we use timers to identify an inactive appender and flush the write behind buffers. A good policy would be to not let any byte in a buffer reach a certain age before being written. The current policy effectively disables this timer if the workload is non-idle (ie we have seen an append within the timeout window---1 sec currently). Clearly there is an edge case here in which a workload writes a small amount of data at a frequency that keeps the timer from firing.

This can be fixed by either not cancelling the timer on append, or tracking the age of bytes in the current chunk and forcing a write before returning from append if a threshold is crossed. In the former case an analysis needs to be completed to to ensure that background write dispatching is safe to interleave with append---it currently does not appear to be.

JIRA Link: CORE-575

developer mode should enable `developer_mode: true` in configuration

We have run into many many tickets regarding this choice.

Let's let developers try the software without exiting due to non-xfs filesystems or under provisioned machines.

unify fetch request handling of partition wrapper

#73 (comment)

Ignore min.insync.replicas if passed as a topic configuration.

I was using the Java Kafka Admin API and was trying to create topic using it but would get back the following:

Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidConfigurationException: Not supported configuration entry
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
    at io.vectorized.App.main(App.java:24)
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: Not supported configuration entry

This was pretty confusing and nothing shows up in the redpanda logs. We should just ignore this field as this is something that many people set when creating topics with Kafka.

Another thing, it would be great to print out an error message in the redpanda log when we are sent a configuration option that we reject. At least this would have helped me in troubleshooting this issue.

It also seems we have issues with the following configs: flush.messages=1 and flush.ms=0

@rkruze anything i'm missing?

CONTRIBUTING.md is out of date

This file doesn't represent the new guidelines

Documentation feedback

I want to track here some of the feedback I already shared with the team:

Single Node Production Setup states XFS as a Requirement, turns out you can totally use redpanda without XFS, this was a show-stopper for me where I'm not looking for the best performance, I just don't want to deal with ZK so a Single node without XFS should work, my recommendation would be to either create a "tier" before "Single Node Production Setup" or the same setup but with two sections: Minimal Requirements(no XFS) and Optimal Requirements(XFS) with a link to the deployment automation. Reference: https://news.ycombinator.com/item?id=25110390
A compatibility table with Kafka would be great, I was specifically interested in topic compaction and retention and couldn't find any mention or info around it [0] https://news.ycombinator.com/item?id=25113978
Apparently packagecloud is being phased out but there are still some references in the documentation, reference https://vectorized.io/documentation (spotted by Alex, I had no clue)

Redpanda not starting

Hello!

I'm trying to do some testing with redpanda. Installed on a AWS instance, Ubuntu 18.04.
I get this in the logs
syschecks - Memory: '499122176' below recommended: '1073741824'

and fails to start.

Yet:

vmstat -s
      2032776 K total memory
       132520 K used memory

What are the minimal system requirements ?

avoid collectives when explicit sequencing is required

let's avoid collectives when we require a specific shutdown ordering.

        -return _gate.close().then([this]() { return _transport.stop(); });
        +return ss::when_all(_gate.close(), _transport.stop()).discard_result();

6df4a48#diff-975c462fbda832b71e7a4f92f664a80f8838b39455d496e46a5635af1d56e054R55

Do not stop recovery STM when there are only relaxed consistency appends

To limit latency we should trigger allow recovery STM sending data to followers immediately after they will be append to the leader log.

kafka: support dynamic quotas

Today we limit based on a per client id, which is a single setting for the whole server. It would be ideal to set a quota on a per-user basis on either byte, requests, or network throughput. This would be similar to how quotas are set in Kafka today. https://kafka.apache.org/documentation/#design_quotasconfig

Some examples of how this is set in Kafka can be found here: https://github.com/JNachtwey/kafka-cheat-sheets/blob/master/kafka-quotas.md

JIRA Link: CORE-576

assertion failure inside abseil flat hash set

on clang/debug receiving this running pandaproxy_client_fixture_rpunit. potentially an issue with lifetime in proxy client but also popped up after merging fetch sessions.

DEBUG 2020-11-25 20:46:55,272 [shard 0] pandaproxy::client - client.cc:75 - updating metadata
TRACE 2020-11-25 20:46:55,287 [shard 1] kafka - requests.cc:87 - Processing name:fetch, key:1, verion:10 for test_client
INFO  2020-11-25 20:46:55,293 [shard 1] kafka - fetch_session_cache.cc:104 - fetch session created: 268435456
TRACE 2020-11-25 20:46:55,299 [shard 1] kafka - fetch_request.cc:405 - fetch reader {storage::log_reader. config {start_offset:{0}, max_offset:{372}, min_bytes:0, max_bytes:1024, type_filter:{}, first_timestamp:nullopt}}
TRACE 2020-11-25 20:46:55,304 [shard 1] kafka - request_context.h:167 - sending 1:fetch response {session_id 268435456 error { error_code: none [0] } partitions {{name {my_topic_0} responses {{id 0 err { error_code: none [0] } high_water -9223372036854775808 last_stable_off -9223372036854775808 aborted {} record_set_len 36739}}}}}
TRACE 2020-11-25 20:46:55,319 [shard 1] kafka - requests.cc:87 - Processing name:metadata, key:3, verion:7 for test_client
TRACE 2020-11-25 20:46:55,322 [shard 1] kafka - request_context.h:167 - sending 3:metadata response throttle_time 0 brokers {id 1 hostname 127.0.0.1 port 9092 rack} cluster_id {nullopt} controller_id 1 topics {err_code { error_code: none [0] } name my_topic_0 is_internal false partitions {err_code { error_code: none [0] } idx 0 leader 1 leader_epoch 0 replicas {{1}} offline {}} tp_aut_ops -1327551456} cluster_aut_ops 0
DEBUG 2020-11-25 20:46:55,331 [shard 1] rpc - server.cc:89 - kafka rpc protocol Closing: 127.0.0.1:49992
pandaproxy_client_fixture_rpunit: v_deps_install/include/absl/container/internal/raw_hash_set.h:128: absl::container_internal::probe_seq<16>::probe_seq(size_t, size_t) [Width = 16]: Assertion `((mask + 1) & mask) == 0 && "not a mask"' failed.

Java Kafka Client fails to connect to docker image

I spun up an instance of vectorized/redpanda:latest and tried connecting from the my Docker host using the standard Java client. This is what I see in my application log:

Connection to node 1 (/0.0.0.0:9092) could not be established.

I believe this is due to the fact that the Java client connects to redpanda as a bootsrtap and as part of the Kafka protocol handshake it is give the address to the actual broker which in this case seems to resolve to 0.0.0.0.

This is from the docker logs:

kafka                  | INFO  2020-11-18 05:31:30,439 [shard 0] redpanda::main - application.cc:162 - advertised_kafka_api:{nullopt}	- Address of Kafka API published to the clients
....
kafka                  | INFO  2020-11-18 05:31:30,440 [shard 0] redpanda::main - application.cc:162 - kafka_api:{host: 0.0.0.0, port: 9092}	- Address and port of an interface to listen for Kafka API requests

I'm migrating my app from Apache Kafka and for my existing config I had to specify some environment variables to get the bootstrap process to work including:

KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093

Note the support for multiple ports mapping to different host names. This was needed to work in both docker compose where the host name for the kafka container was literally kafka and when connecting from the host using an exposed port.

Does your Docker image have support for similar ENV configs? The redpanda docker image on Docker Hub doesn't have a README yet (I know you're working on one).

A docker public image

Hello, I would like to try easily redpanda as a drop-in kafka alternative.

A public docker image in the docker hub would be great ( as what is doing confluent : https://hub.docker.com/r/confluentinc/cp-kafka/ )

So I can try it in a docker-compose like this :

version: '3.8'

services:

  zookeeper:
    image: confluentinc/cp-zookeeper:6.0.0
    ports:
      - 2181:2181
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka-broker:
    image: confluentinc/cp-kafka:6.0.0
    ports:
      - 9092:9092
    expose:
      - 29092
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-broker:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

  schema-registry:
    image: confluentinc/cp-schema-registry:6.0.0
    hostname: schema-registry
    depends_on:
      - kafka-broker
      - zookeeper
    ports:
      - 8081:8081
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: zookeeper:2181

I know zookepper will no longer be needed.

Thank you

MacOS: Homebrew formula

For easier installation on MacOS, a homebrew formula would be great. Is this already planned?

rpk/config: Make redpanda.advertised* optional

The redpanda.advertised_kafka_api and redpanda.advertised_rpc_server are optional for redpanda, but rpk doesn't treat them as such. Therefore, if they're not set by the user, the next time rpk writes the config they will be persisted with an empty address and 0 port, which will make any client connection fail.

storage: remove flush requirement from truncation

Truncation flushes the underlying file descriptor, which is not required in general. The caller may flush if needed. Since truncation may be common in several raft-related scenarios, this may be an important optimization.

eliminate dedicated write-behind buffering for each segment appender

per-core shared chunk cache
minimum pre-allocate reserve (e.g. 10%)
some reasonable upper limit (e.g. 30%)
remove or expand per segment limit (currently at 1mb outstanding)

Multiple listeners support : Confluent Schema-Registry with redpanda

I want to use the confluent schema-registry with redpanda , I'm trying with the confluent docker image and I don't know what to set for :

SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL

From this blog post https://www.confluent.fr/blog/how-to-prepare-for-kip-500-kafka-zookeeper-removal-guide/ , we should replace

kafkastore.connection.url by kafkastore.bootstrap.servers

So SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL become SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS

https://docs.confluent.io/current/installation/docker/config-reference.html#schema-registry-configuration

Docker-compose configuration :

version: '3.8'

services:

  kafka-broker:
    image: vectorized/redpanda
    ports:
      - 9092:9092
    expose:
      - 29092
    environment:
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-broker:29092,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT

  schema-registry:
    image: confluentinc/cp-schema-registry:6.1.0
    hostname: schema-registry
    depends_on:
      - kafka-broker
    ports:
      - 8081:8081
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: kafka-broker:29092

The shema-registry container logs :

[main] ERROR io.confluent.admin.utils.ClusterStatus - Error while getting broker list.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1605548460336, tries=1, nextAllowedTryMs=1605548460437) timed out at 1605548460337 after 1 attempt(s)
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:149)
	at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:150)
Caused by: org.apache.kafka.common.errors.TimeoutException: Call(callName=listNodes, deadlineMs=1605548460336, tries=1, nextAllowedTryMs=1605548460437) timed out at 1605548460337 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment.
[main] INFO io.confluent.admin.utils.ClusterStatus - Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
[main] ERROR io.confluent.admin.utils.ClusterStatus - Expected 1 brokers but found only 0. Brokers found [].

and Redpanda logs :

Missing license key. Please get a new one at https://vectorized.io/download-trial/
Error fetching the Redpanda topic details: dial tcp :0: connect: connection refused
  Version                                 release-20.11.1 (rev 0fb1a90)                                        
  OS                                      x86_64                                                            5.4.0-53-generic                                                  #59-Ubuntu SMP Wed Oct 21 09:38:44 UTC 2020

Thank you

Introduce an HTTP endpoint for healthchecks

Should be compatible with Kubernetes probes

pp/handlers: Parameter validation and error handling

There are a few times of parameters, query, body, path.

Improve validation and error handling.

Passing a non-integer results in something like:

{"message": "boost::wrapexcept<boost::bad_lexical_cast> (bad lexical cast: source type value could not be interpreted as target)", "code": 500}

storage: adaptive fallocate should not wait on all inflight writes

When fallocating segment capacity the fallocation routine inserts a write barrier and waits for all inflight writes to complete. This was a non-issue when the cap on inflight writes was small. But now this cap has been lifted, and this barrier will introduce unnecessary latency. We still need to wait on inflight writes, but we can limit the set of writes to inflight writes to the tail that intersect the fallocation range.

JIRA Link: CORE-574

storage: delay chunk hydration until required

On truncation, the head chunk is rehydrated. However, this isn't strictly necessary if no additional appends will occur. Therefore, we can avoid the extra read by delaying rehydration until a subsequent operation that requires it, like append.

allocation underflow assertion triggered

Setup:

7 nodes cluster
128 partitions
16 bytes key (keys carnality 10mm)
4k payload
compacted topic verifier.

Error was triggered on one of the nodes after ~30h.

 (inlined by) seastar::backtrace_buffer::append_backtrace() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:731
 (inlined by) seastar::print_with_backtrace(seastar::backtrace_buffer&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:752
 (inlined by) seastar::print_with_backtrace(char const*) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:759
 (inlined by) seastar::sigabrt_action() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3486
 (inlined by) operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3468
 (inlined by) __invoke at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3464
0x000029cbde29da8f: ?? ??:0
0x0000000000000000: ?? ??:0
0x0000000000000000: ?? ??:0
0x000000000133b9d7: seastar::memory::abort_on_underflow(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1203
 (inlined by) seastar::memory::allocate_large(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1208
 (inlined by) seastar::memory::allocate(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1269
0x0000000001347245: malloc at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/memory.cc:1608
0x0000000000be61b6: temporary_buffer at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/temporary_buffer.hh:73
 (inlined by) seastar::input_stream<char>::read_exactly(unsigned long) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/iostream-impl.hh:190
0x0000000000be46aa: operator() at /home/mmaslanka/dev/v/build/release/clang/../../../src/v/kafka/protocol_utils.cc:33
0x0000000000be2d5c: seastar::future<std::__1::optional<kafka::request_header> > seastar::futurize<seastar::future<std::__1::optional<kafka::request_header> > >::invoke<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::temporary_buffer<char> >(kafka::parse_header(seastar::input_stream<char>&)::$_0&&, seastar::temporary_buffer<char>&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:2135
 (inlined by) seastar::future<std::__1::optional<kafka::request_header> > seastar::future<seastar::temporary_buffer<char> >::then_impl<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::future<std::__1::optional<kafka::request_header> > >(kafka::parse_header(seastar::input_stream<char>&)::$_0&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1601
 (inlined by) seastar::internal::future_result<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::temporary_buffer<char> >::future_type seastar::internal::call_then_impl<seastar::future<seastar::temporary_buffer<char> > >::run<kafka::parse_header(seastar::input_stream<char>&)::$_0>(seastar::future<seastar::temporary_buffer<char> >&, kafka::parse_header(seastar::input_stream<char>&)::$_0&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1234
 (inlined by) seastar::future<std::__1::optional<kafka::request_header> > seastar::future<seastar::temporary_buffer<char> >::then<kafka::parse_header(seastar::input_stream<char>&)::$_0, seastar::future<std::__1::optional<kafka::request_header> > >(kafka::parse_header(seastar::input_stream<char>&)::$_0&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1520
 (inlined by) kafka::parse_header(seastar::input_stream<char>&) at /home/mmaslanka/dev/v/build/release/clang/../../../src/v/kafka/protocol_utils.cc:9
0x0000000000abf00e: operator() at /home/mmaslanka/dev/v/build/release/clang/../../../src/v/kafka/protocol.cc:60
 (inlined by) _ZNSt3__18__invokeIRZN5kafka8protocol18connection_context19process_one_requestEvE3$_4JNS_8optionalImEEEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS8_DpOS9_ at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/type_traits:3539
 (inlined by) std::__1::invoke_result<kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long> >::type std::__1::invoke<kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long> >(kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long>&&) at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/functional:2902
 (inlined by) auto seastar::internal::future_invoke<kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long> >(kafka::protocol::connection_context::process_one_request()::$_4&, std::__1::optional<unsigned long>&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1211
 (inlined by) operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1582
 (inlined by) void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<std::__1::optional<unsigned long> >::then_impl_nrvo<kafka::protocol::connection_context::process_one_request()::$_4, seastar::future<void> >(kafka::protocol::connection_context::process_one_request()::$_4&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&, seastar::future_state<std::__1::optional<unsigned long> >&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&, seastar::future_state<std::__1::optional<unsigned long> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:2120
 (inlined by) operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:1575
 (inlined by) seastar::continuation<seastar::internal::promise_base_with_type<void>, kafka::protocol::connection_context::process_one_request()::$_4, seastar::future<std::__1::optional<unsigned long> >::then_impl_nrvo<kafka::protocol::connection_context::process_one_request()::$_4, seastar::future<void> >(kafka::protocol::connection_context::process_one_request()::$_4&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, kafka::protocol::connection_context::process_one_request()::$_4&, seastar::future_state<std::__1::optional<unsigned long> >&&)#1}, std::__1::optional<unsigned long> >::run_and_dispose() at /home/mmaslanka/dev/v/build/release/clang/v_deps_install/include/seastar/core/future.hh:767
0x00000000013cd14f: seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2198
 (inlined by) seastar::reactor::run_some_tasks() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2577
0x00000000013cfd49: seastar::reactor::run() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:2732
0x000000000142bc41: operator() at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/reactor.cc:3908
0x000000000136430f: std::__1::__function::__value_func<void ()>::operator()() const at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/functional:1867
 (inlined by) std::__1::function<void ()>::operator()() const at /home/mmaslanka/dev/v/build/llvm/llvm-bin/bin/../include/c++/v1/functional:2473
 (inlined by) seastar::posix_thread::start_routine(void*) at /home/mmaslanka/dev/v/build/release/clang/v_deps_build/seastar-prefix/src/seastar/src/core/posix.cc:60

make code formatting tooling available in redpanda repository

Ultimately we are going to ask external contributors to ensure their code is properly formatted. It may be convenient to give them the same tools we use to 1. manually apply formatting and 2. check that their formatting is correct.

use named batch types instead of bare ids

0d1beeb#diff-fcd7fbf0643f830200a530fd291bf91d7c4a2b0919bffbfcea623a04ddd046ccR350

script_manager_unit test failing

This doesn't happen very often. Probably why no one has observed it yet. Test crashes, on test 'test_copro_topics', stack dump as follows:

INFO  2020-11-23 15:16:56,394 [shard 10] raft - [group_id:25, {kafka/foo/4}] vote_stm.cc:234 - became the 
leader term:1                                                                                             
INFO  2020-11-23 15:16:56,394 [shard 3] coproc - router.cc:344 - Inserted ntp {kafka/baz/2} id 1523       
INFO  2020-11-23 15:16:56,394 [shard 5] coproc - router.cc:344 - Inserted ntp {kafka/baz/4} id 1523       
INFO  2020-11-23 15:16:56,394 [shard 3] coproc - router.cc:344 - Inserted ntp {kafka/baz/17} id 1523      
INFO  2020-11-23 15:16:56,394 [shard 11] coproc - router.cc:344 - Inserted ntp {kafka/baz/10} id 1523     
INFO  2020-11-23 15:16:56,395 [shard 10] coproc - router.cc:344 - Inserted ntp {kafka/baz/9} id 1523      
INFO  2020-11-23 15:16:56,395 [shard 8] coproc - router.cc:344 - Inserted ntp {kafka/baz/7} id 1523       
INFO  2020-11-23 15:16:56,395 [shard 12] coproc - router.cc:344 - Inserted ntp {kafka/baz/11} id 1523     
=================================================================                                         
INFO  2020-11-23 15:16:56,395 [shard 14] coproc - router.cc:344 - Inserted ntp {kafka/baz/13} id 1523     
==3129307==ERROR: AddressSanitizer: heap-use-after-free on address 0x6120000a9138 at pc 0x56523513e3fa bp 
0x7f17575b5500 sp 0x7f17575b54f8                                                                          
INFO  2020-11-23 15:16:56,395 [shard 4] coproc - router.cc:344 - Inserted ntp {kafka/baz/3} id 1523       
INFO  2020-11-23 15:16:56,395 [shard 1] coproc - router.cc:344 - Inserted ntp {kafka/baz/15} id 1523      
INFO  2020-11-23 15:16:56,395 [shard 6] coproc - router.cc:344 - Inserted ntp {kafka/baz/5} id 1523       
READ of size 8 at 0x6120000a9138 thread T23 (reactor-12)                                                  
INFO  2020-11-23 15:16:56,395 [shard 1] coproc - router.cc:344 - Inserted ntp {kafka/baz/0} id 1523       
INFO  2020-11-23 15:16:56,395 [shard 2] coproc - router.cc:344 - Inserted ntp {kafka/baz/1} id 1523       
INFO  2020-11-23 15:16:56,395 [shard 7] coproc - router.cc:344 - Inserted ntp {kafka/baz/6} id 1523       
INFO  2020-11-23 15:16:56,395 [shard 2] coproc - router.cc:344 - Inserted ntp {kafka/baz/16} id 1523      
INFO  2020-11-23 15:16:56,395 [shard 9] coproc - router.cc:344 - Inserted ntp {kafka/baz/8} id 1523       
    #0 0x56523513e3f9  (/home/robert/workspace/redpanda/vbuild/debug/clang/bin/coproc_unit_tests_rpunit+0x
12d503f9)                                                                                                 
INFO  2020-11-23 15:16:56,396 [shard 13] coproc - router.cc:344 - Inserted ntp {kafka/baz/12} id 1523     
    #1 0x56523513e001  (/home/robert/workspace/redpanda/vbuild/debug/clang/bin/coproc_unit_tests_rpunit+0$

Build failures due to missing "ensurepip"

[14/71] Performing install step for 'kafka-python-pex'
FAILED: kafka-python-pex-prefix/src/kafka-python-pex-stamp/kafka-python-pex-install
cd /home/ubuntu/rp3/build/deps_build/kafka-python-pex-prefix/src/kafka-python-pex-build && /home/ubuntu/cmake-3.12.0-Linux-x86_64/bin/cmake -E chdir /home/ubuntu/rp3/build/deps_build/kafka-python-pex-prefix/src/kafka-python-pex python3 -m venv env && /home/ubuntu/rp3/build/deps_build/kafka-python-pex-prefix/src/kafka-python-pex/env/bin/pip install pex && /home/ubuntu/rp3/build/deps_build/kafka-python-pex-prefix/src/kafka-python-pex/env/bin/pex kafka-python -o /home/ubuntu/rp3/build/deps_install/bin/kafka-python-env && /home/ubuntu/cmake-3.12.0-Linux-x86_64/bin/cmake -E touch /home/ubuntu/rp3/build/deps_build/kafka-python-pex-prefix/src/kafka-python-pex-stamp/kafka-python-pex-install
The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

apt-get install python3-venv`

It went away once I installed python3.6-venv (ie modified this line from install-dependencies.sh)

rpk on macOS should disable unsupported commands

As pointed out in #165 some commands like rpk status doesn't work as expected on macOS. The same is true for commands like rpk config. These commands should be disabled until they are fully supported.

Using rpk on osx is a bit confusing. One error suggests that I need a license.

I'm really hoping to be one of your first cloud customers but I need to get the project done first. I'm assuming the licensing error is legacy, but I just want to be sure (I'm basing my speculation on https://vectorized.io/open-source/).

I was fumbling around trying to figure out how to create partitions so I could test my existing kafka-based tests against it. I create all topics programmatically. So far as I can tell, sarama is able to connect and is just failing silently on creating topics due to lack of partitions. I'm just not sure how to go about creating those :)

To highlight the error that you may be most interested in, I've extracted that snippet out from the longer saga of me fumbling through the command line.

▶ cd ~/.rpk/cluster/node-0/conf
▶ rpk status
Missing license key. Please get a new one at https://vectorized.io/download-trial/
Error querying OS info: uname not available in MacOS
Error initializing redpanda client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws.com/309512982/f2d18e00-290f-11eb-990a-47764a
######################################################################## 100.0%
==> Caveats
Redpanda - The fastest queue in the west!

This installs RPK which, with Docker, enables the running of a local cluster
for testing purposes.

You can start a 3 node cluster locally using the following command:

    rpk container start -n 3

You can then interact with the cluster using commands like the following:

    rpk api status

When done, you can stop and delete the cluster with the following command:

    rpk container purge

For information on how to setup production evironments, check out our
installation guide here: https://vectorized.io/documentation/setup-guide/
==> Summary
🍺  /usr/local/Cellar/redpanda/20.11.4: 3 files, 24.2MB, built in 4 seconds
==> `brew cleanup` has not been run in 30 days, running now...
# truncated
Downloading latest version of Redpanda
Starting cluster
  NODE ID  ADDRESS          CONFIG
  0        127.0.0.1:50333  /Users/chance/.rpk/cluster/node-0/conf/redpanda.yaml

Cluster started! You may use 'rpk api' to interact with the cluster. E.g:

rpk api status

▶ rpk api status
  Redpanda Cluster Status

  0 (127.0.0.1:50333)      (No partitions)

▶ rpk mode
Error: requires a mode [dev, development, prod, production]


▶ rpk mode dev
Error: Open /Users/chance/Projects/redpanda.yaml, /etc/redpanda/redpanda.yaml, /Users/chance/redpanda.yaml, /Users/redpanda.yaml: file does not exist

▶ rpk mode development
Error: Open /Users/chance/redpanda.yaml, /etc/redpanda/redpanda.yaml, /Users/redpanda.yaml: file does not exist

▶ rpk status
Error: Open /Users/chance/redpanda.yaml, /etc/redpanda/redpanda.yaml, /Users/redpanda.yaml: file does not exist


▶ rpk config bootstrap --id 0
Error: YAML

▶ sudo rpk config bootstrap --id 0
Password:
Error: YAML

▶    rpk container start -n 3

Found an existing cluster:

  NODE ID  ADDRESS           CONFIG
  0        172.24.1.2:50333  /Users/chance/.rpk/cluster/node-0/conf/redpanda.yaml

▶ cat /Users/chance/.rpk/cluster/node-0/conf/redpanda.yaml
config_file: /etc/redpanda/redpanda.yaml
node_uuid: QS89uT3jmzYcd1CVWLMjqzyr52LuZRRQQ3a2EEcJTGSdHdSyX
redpanda:
  admin:
    address: 0.0.0.0
    port: 9644
  advertised_kafka_api:
    address: 127.0.0.1
    port: 50333
  advertised_rpc_api:
    address: 172.24.1.2
    port: 33145
  data_directory: /var/lib/redpanda/data
  developer_mode: true
  kafka_api:
    address: 172.24.1.2
    port: 9092
  kafka_api_tls:
    cert_file: ""
    enabled: false
    key_file: ""
    truststore_file: ""
  node_id: 0
  rpc_server:
    address: 172.24.1.2
    port: 33145
  seed_servers: []
rpk:
  coredump_dir: /var/lib/redpanda/coredump
  enable_memory_locking: false
  enable_usage_stats: false
  overprovisioned: true
  tls:
    cert_file: ""
    key_file: ""
    truststore_file: ""
  tune_aio_events: false
  tune_clocksource: false
  tune_coredump: false
  tune_cpu: false
  tune_disk_irq: false
  tune_disk_nomerges: false
  tune_disk_scheduler: false
  tune_fstrim: false
  tune_network: false
  tune_swappiness: false
  tune_transparent_hugepages: false


▶ cd ~/.rpk/cluster/node-0/conf

▶ rpk status
Missing license key. Please get a new one at https://vectorized.io/download-trial/
Error querying OS info: uname not available in MacOS
Error initializing redpanda client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)
  Version                                 release-20.11.4 (rev e6f825d46565f62db99c52f82dce2bc805bc1ba6)
  CPU Model                               Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
  config_file                             /etc/redpanda/redpanda.yaml
  node_uuid                               QS89uT3jmzYcd1CVWLMjqzyr52LuZRRQQ3a2EEcJTGSdHdSyX
  redpanda.admin                          0.0.0.0:9644
  redpanda.data_directory                 /var/lib/redpanda/data
  redpanda.developer_mode                 true
  redpanda.kafka_api                      172.24.1.2:9092
  redpanda.kafka_api_tls.cert_file
  redpanda.kafka_api_tls.enabled          false
  redpanda.kafka_api_tls.key_file
  redpanda.kafka_api_tls.truststore_file
  redpanda.node_id                        0
  redpanda.rpc_server                     172.24.1.2:33145
  rpk.coredump_dir                        /var/lib/redpanda/coredump
  rpk.enable_memory_locking               false
  rpk.enable_usage_stats                  false
  rpk.overprovisioned                     true
  rpk.tls.cert_file
  rpk.tls.key_file
  rpk.tls.truststore_file
  rpk.tune_aio_events                     false
  rpk.tune_clocksource                    false
  rpk.tune_coredump                       false
  rpk.tune_cpu                            false
  rpk.tune_disk_irq                       false
  rpk.tune_disk_nomerges                  false
  rpk.tune_disk_scheduler                 false
  rpk.tune_fstrim                         false
  rpk.tune_network                        false
  rpk.tune_swappiness                     false
  rpk.tune_transparent_hugepages          false

storage: attach reclaimer to segment appender chunk cache

Currently the seastar memory reclaimer will reclaim memory from our batch cache in low memory situations. The chunk cache also makes a large memory reservation on bootup, and we could reclaim memory from the chunk cache in low memory situations.

This is low priority for now we haven't been seeing OOMs recently but may be some easy way to get some extra memory later if we start to see some memory pressure issues.

rpk: Allow passing arbitrary config entries to `rpk api topic create`

MacOS: Error message when Docker is not running misleading

On MacOS, when Docker is not running, the following error message is returned:

$ rpk container start
Downloading latest version of Redpanda
Error: Couldn't pull image and a local one wasn't found either.
Please check your internet connection and try again.

It should be easy to detect if Docker is not running and let the user know.

materialized topics should minimize flushing

Flushing is expensive, and the only time I am imagining where it would be required is on shutdown.

cbf36dd#diff-fcd7fbf0643f830200a530fd291bf91d7c4a2b0919bffbfcea623a04ddd046ccR268

0d1beeb#diff-fcd7fbf0643f830200a530fd291bf91d7c4a2b0919bffbfcea623a04ddd046ccR251

Replication should be rack-aware

It should be possible to configure replicas to exist on different racks or availability zones.

specialize CODEOWNERS for co-proc javascript tree

this is useful for custom tailoring the set of automatically selected code reviewers

Configuration should be stored in Redpanda

All configuration for Redpanda is currently stored in configuration files normally stored in the /etc/redpanda directory. Ideally, most of the configuration options needed should be stored in Redpanda itself. This would make it possible to quickly push out configuration information to the whole cluster. This could be done with something like a compacted topic.

One could envision just a list of seed servers, advertised address, and data directory as the only configuration options local to the instance, with everything else being global and saved in Redpanda centrally. It would then make sense to use something like rpk to configure the parameters stored in Redpanda.