Library kfkp
implements a kafka producer pool, which is a high-performance Go package designed to efficiently manage Kafka producer connections.
-
Dynamic Scaling: Automatically manages the creation, acquisition, release, and destruction of Kafka producer connections, with support for dynamic scaling based on demand.
-
Synchronization Mechanisms: Utilizes synchronization mechanisms to handle connection contention and timeout scenarios, ensuring efficient and reliable connection management.
-
Configuration Options: Implements the Options pattern to configure pool parameters, allowing fine-tuning of the connection pool behavior according to application needs.
-
Timeout Queue Mechanism: Supports a timeout waiting queue mechanism to handle connection requests and timeouts effectively.
-
Connection Cleanup: Periodically cleans up long-unused connections to maintain optimal performance and resource utilization.
go get -u install "github.com/Sh-Fang/kfkp"
package main
import (
"log"
"strconv"
"github.com/Sh-Fang/kfkp"
"github.com/segmentio/kafka-go"
)
func main() {
pool, err := kfkp.NewPool(
kfkp.WithInitCapacity(10),
kfkp.WithMaxCapacity(100),
kfkp.WithMaxIdle(10),
kfkp.WithBrokerAddress("localhost:9092"),
kfkp.WithTopic("topic"),
kfkp.WithRequiredAcks(kafka.RequireNone),
kfkp.WithAsync(false),
)
if err != nil {
log.Fatal(err)
}
pd, err := pool.GetConn()
if err != nil {
return
}
err = pd.SendMessage([]byte("123"), []byte(strconv.Itoa(111)))
if err != nil {
return
}
err = pool.PutConn(pd)
if err != nil {
return
}
err = pool.ClosePool()
if err != nil {
log.Fatal(err)
}
}
You can invoke the NewPool method to instantiate a pool with options, as follows:
pool, err := kfkp.NewPool(
kfkp.WithInitCapacity(10),
kfkp.WithMaxCapacity(100),
kfkp.WithMaxIdle(10),
kfkp.WithBrokerAddress("localhost:9092"),
kfkp.WithTopic("topic"),
kfkp.WithRequiredAcks(kafka.RequireNone),
kfkp.WithAsync(false),
)
Connection can be getting from pool by calling pool.GetConn()
pd, err := pool.GetConn()
Connection can be putting back to pool by calling pool.PutConn(pd)
err = pool.PutConn(pd)
You can close the connection pool and release all the resources.
err = pool.ClosePool()
The source code in kfkp is available under the MIT License.
- ants: A powerful and reliable pooling solution for Go.