rkruze / franz-go

franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.8.0+. Producing, consuming, transacting, administrating, etc.

franz-go - A complete Apache Kafka client written in Go

Franz-go is an all-encompassing Apache Kafka client fully written Go. This library aims to provide every Kafka feature from Apache Kafka v0.8.0 onward. It has support for transactions, regex topic consuming, the latest partitioning strategies, data loss detection, closest replica fetching, and more. If a client KIP exists, this library aims to support it.

This library attempts to provide an intuitive API while interacting with Kafka the way Kafka expects (timeouts, etc.).

Features

• Feature complete client (Kafka >= 0.8.0 through v2.8.0+)
• Full Exactly-Once-Semantics (EOS)
• Idempotent & transactional producers
• Simple (legacy) consumer
• Group consumers with eager (roundrobin, range, sticky) and cooperative (cooperative-sticky) balancers
• All compression types supported: gzip, snappy, lz4, zstd
• SSL/TLS provided through custom dialer options
• All SASL mechanisms supported (GSSAPI/Kerberos, PLAIN, SCRAM, and OAUTHBEARER)
• Low-level admin functionality supported through a simple Request function
• Utilizes modern & idiomatic Go (support for contexts, variadic configuration options, ...)
• Highly performant, see Performance (benchmarks will be added)
• Written in pure Go (no wrapper lib for a C library or other bindings)
• Ability to add detailed log messages or metrics using hooks

Getting started

Here's a basic overview of producing and consuming:

seeds := []string{"localhost:9092"}
client, err := kgo.NewClient(kgo.SeedBrokers(seeds...))
if err != nil {
	panic(err)
}
defer client.Close()

ctx := context.Background()

// 1.) Producing a message
// All record production goes through Produce, and the callback can be used
// to allow for synchronous or asynchronous production. ProduceSync exists if
// you always want to produce synchronously.
var wg sync.WaitGroup
wg.Add(1)
record := &kgo.Record{Topic: "foo", Value: []byte("bar")}
err := client.Produce(ctx, record, func(_ *Record, err error) {
	defer wg.Done()
	if err != nil {
		fmt.Printf("record had a produce error: %v\n", err)
	}

})
if err != nil {
	panic("we are unable to produce if the context is canceled, we have hit max buffered, " +
		"or if we are transactional and not in a transaction")
}
wg.Wait()

// 2.) Consuming messages from a topic
// Consuming can either be direct (no consumer group), or through a group. Below, we use a group.
client.AssignGroup("my-group-identifier", kgo.GroupTopics("foo"))
for {
	fetches := client.PollFetches(ctx)
	if errs := fetches.Errors(); len(errs) > 0 {
		// All errors are retried internally when fetching, but non-retriable errors are
		// returned from polls so that users can notice and take action.
		panic(fmt.Sprint(errs))
	}

	// We can iterate through a record iterator...
	iter := fetches.RecordIter()
	for !iter.Done() {
		record := iter.Next()
		fmt.Println(string(record.Value), "from an iterator!")
	}

	// or a callback function.
	fetches.EachPartition(func(p kgo.FetchTopicPartition) {
		for _, record := range p.Partition.Records {
			fmt.Println(string(record.Value), "from range inside a callback!")
		}

		// We can even use a second callback!
		p.EachRecord(func(record *Record) {
			fmt.Println(string(record.Value), "from a second callback!")
		})
	})
}

This only shows producing and consuming in the most basic sense, and does not show the full list of options to customize how the client runs, nor does it show transactional producing / consuming. Check out the examples directory for more!

API reference documentation can be found on . Supplementary information can be found in the docs directory:

docs
+-- admin requests for an overview of how to issue admin requests
+-- architecture for descrbing the packages in franz-go and some internals
+-- consuming for a description of consuming in a group (and a short section on the simple consumer)
+-- producing for a description of producing and producing guarantees
+-- performance for some notes about performance

Version Pinning

By default, the client issues an ApiVersions request on connect to brokers and defaults to using the maximum supported version for requests that each broker supports.

Kafka 0.10.0 introduced the ApiVersions request; if you are working with brokers older than that, you must use the kversions package. Use the MaxVersions option for the client if you do so.

As well, it is recommended to set the MaxVersions to the version of your broker cluster. Until KIP-584 is implemented, it is possible that if you do not pin a max version, this client will speak with some features to one broker while not to another when you are in the middle of a broker update roll.

Metrics & logging

The franz-go client takes a neutral approach to metrics by providing hooks that you can use to plug in your own metrics.

All connections, disconnections, reads, writes, and throttles can be hooked into. If there is an aspect of the library that you wish you could have insight into, please open an issue and we can discuss adding another hook.

Hooks allow you to log in the event of specific errors, or to trace latencies, count bytes, etc., all with your favorite monitoring systems.

In addition to hooks, logging can be plugged in with a general Logger interface. A basic logger is provided if you just want to write to a given file in a simple format. All logs have a message and then key/value pairs of supplementary information. It is recommended to always use a logger and to use LogLevelInfo.

See this example for an example of integrating with prometheus!

Supported KIPs

Theoretically, this library supports every (non-Java-specific) client facing KIP. Most are tested, some need testing. Any KIP that simply adds or modifies a protocol is supported by code generation.

• KIP-12 (sasl & ssl; 0.9.0)
• KIP-13 (throttling; supported but not obeyed)
• KIP-31 (relative offsets in message set; 0.10.0)
• KIP-32 (timestamps in message set v1; 0.10.0)
• KIP-35 (adds ApiVersion; 0.10.0)
• KIP-36 (rack aware replica assignment; 0.10.0)
• KIP-40 (ListGroups and DescribeGroup v0; 0.9.0)
• KIP-43 (sasl enhancements & handshake; 0.10.0)
• KIP-54 (sticky group assignment)
• KIP-62 (join group rebalnce timeout, background thread heartbeats; v0.10.1)
• KIP-74 (fetch response size limit; 0.10.1)
• KIP-78 (cluster id in metadata; 0.10.1)
• KIP-79 (list offset req/resp timestamp field; 0.10.1)
• KIP-84 (sasl scram; 0.10.2)
• KIP-98 (EOS; 0.11.0)
• KIP-101 (offset for leader epoch introduced; broker usage yet; 0.11.0)
• KIP-107 (delete records; 0.11.0)
• KIP-108 (validate create topic; 0.10.2)
• KIP-110 (zstd; 2.1.0)
• KIP-112 (JBOD disk failure, protocol changes; 1.0.0)
• KIP-113 (JBOD log dir movement, protocol additions; 1.0.0)
• KIP-124 (request rate quotas; 0.11.0)
• KIP-133 (describe & alter configs; 0.11.0)
• KIP-152 (more sasl, introduce sasl authenticate; 1.0.0)
• KIP-183 (elect preferred leaders; 2.2.0)
• KIP-185 (idempotent is default; 1.0.0)
• KIP-195 (create partitions request; 1.0.0)
• KIP-207 (new error in list offset request; 2.2.0)
• KIP-219 (throttling happens after response; 2.0.0)
• KIP-226 (describe configs v1; 1.1.0)
• KIP-227 (incremental fetch requests; 1.1.0)
• KIP-229 (delete groups request; 1.1.0)
• KIP-255 (oauth via sasl/oauthbearer; 2.0.0)
• KIP-279 (leader / follower failover; changed offsets for leader epoch; 2.0.0)
• KIP-320 (fetcher log truncation detection; 2.1.0)
• KIP-322 (new error when delete topics is disabled; 2.1.0)
• KIP-339 (incremental alter configs; 2.3.0)
• KIP-341 (sticky group bug fix)
• KIP-342 (oauth extensions; 2.1.0)
• KIP-345 (static group membership, see KAFKA-8224)
• KIP-360 (safe epoch bumping for UNKNOWN_PRODUCER_ID; 2.5.0)
• KIP-368 (periodically reauth sasl; 2.2.0)
• KIP-369 (always round robin produce partitioner; 2.4.0)
• KIP-380 (inter-broker command changes; 2.2.0)
• KIP-392 (fetch request from closest replica w/ rack; 2.2.0)
• KIP-394 (require member.id for initial join; 2.2.0)
• KIP-412 (dynamic log levels with incremental alter configs; 2.4.0)
• KIP-429 (incremental rebalance, see KAFKA-8179; 2.4.0)
• KIP-430 (include authorized ops in describe groups; 2.3.0)
• KIP-447 (transaction changes to better support group changes; 2.5.0)
• KIP-455 (admin replica reassignment; 2.4.0)
• KIP-460 (admin leader election; 2.4.0)
• KIP-464 (defaults for create topic; 2.4.0)
• KIP-467 (produce response error change for per-record errors; 2.4.0)
• KIP-480 (sticky partition producing; 2.4.0)
• KIP-482 (tagged fields; KAFKA-8885; 2.4.0)
• KIP-496 (offset delete admin command; 2.4.0)
• KIP-497 (new API to alter ISR; 2.7.0)
• KIP-498 (add max bound on reads; unimplemented in Kafka)
• KIP-511 (add client name / version in apiversions req; 2.4.0)
• KIP-518 (list groups by state; 2.6.0)
• KIP-525 (create topics v5 returns configs; 2.4.0)
• KIP-526 (reduce metadata lookups; done minus part 2, which we wont do)
• KIP-546 (client quota APIs; 2.5.0)
• KIP-554 (broker side SCRAM API; 2.7.0)
• KIP-559 (protocol info in sync / join; 2.5.0)
• KIP-569 (doc/type in describe configs; 2.6.0)
• KIP-570 (leader epoch in stop replica; 2.6.0)
• KIP-580 (exponential backoff; 2.6.0)
• KIP-588 (producer recovery from txn timeout; 2.7.0)
• KIP-590 (support for forwarding admin requests; 2.7.0)
• KIP-595 (new APIs for raft protocol; 2.7.0)
• KIP-599 (throttle create/delete topic/partition; 2.7.0)
• KIP-664 (describe producers, describe / list transactions; mostly 2.8.0 [write txn markers missing])
• KIP-700 (describe cluster; 2.8.0)