davidsbond / arrebato Goto Github PK

Right now, TLS should work between clients and between raft members with mutual TLS and SPIFFE identities.

Ideally, this should be reviewed over by someone with more knowledge than me on the subject to make sure everything is above board and correct. Then documentation around creating a secure deployment should be made.

If the helm chart already exists from #82 it should also be updated to support all the TLS goodness, preferably with cert manager's CSI driver which seems ideal for this use-case.

Right now, a single consumer works in isolation against a topic. Ideally, we should try and come up with a system where multiple instances of the same consumer can work together to get through a topic quicker. Similar to Kafka's consumer groups.

Below is an approach that could be taken:

1. Assign topics to individual members of the raft quorum and only allow consumers of that topic to get it from that server. For this, we'd probably need additional raft commands when nodes join/leave and commands for assigning topics to nodes. When a server leaves, the topics should be redistributed among the remaining nodes.
2. Advertise the topics available on the describe node call so that clients know where to go
3. Per consumer id/topic combination, have a channel that fans out to all consumers of that topic that share a consumer id
4. Use a hashing algorithm (like crc32) so that messages that share a key are always delivered to the same consumer in the group. This is important for message ordering. Blindly distributing messages among all the consumers could lead to messages being processed out of order

Currently, the client works by providing a list of server addresses. Each server is checked to see if it is the leader node or not. If an API call returns a FAILED_PRECONDITION grpc code then the client attempts to find the leader again and retries the request.

This is a bit flaky and error prone. For example, if full leadership is lost this will infinite loop and probably cause a stack overflow.

The client logic for handling this needs to be better and more robust. Potential ideas:

1. Have clients also use serf. When they connect via serf they indicate via tags that they're a client and not a server. This way, the server does not attempt to add them to the cluster. The client will then receive events as servers are added and removed. With this, the client need only know about one server, and automagically get information on all the other available servers. Every time a server is added or removed, recheck the leadership. When the client makes API requests to the wrong server, just propagate that upwards as it should be very unlikely to occur and the client can manually retry.
2. ...

Currently, the project only provides a kustomize manifest.

1. This isn't particularly useful for customising deployments, you can't template much. So you end up having to do manual changes if you want to use the cert-manager CSI driver for TLS.
2. It's easy to forget to change the image version before you tag a release

Ideally, we should have a helm chart and a CI process that ensures the image version is correct. We can also use helm template in the CI process to produce static manifests for anyone that wants a one-liner.