activesphere / kqm Goto Github PK

Instead of using recover in the double defer blocks to handle a panic while writing to a closed channel, we should use the Done() channel provided by the context to see if the message channel is closed and if it is, write to the error channel so that the function becomes eligible for a retry.

We should use docker-compose to spawn docker containers which will run the tests. Next step to this will be automating the test execution by configuring Travis CI (#4).

Essential components of the system should be placed in different containers instead of a single one. This will enable us to look out for corner cases related to network latencies, network failures etc. We should also use a tool like Toxiproxy to introduce network faults/throttles.

Container configuration should be as follows:

1. KQM and Test Runner should run in a single container. This container's image should have go preinstalled. Let's call this Container A.
2. Kafka should run in a separate container. This container's image should have 'apache-kafka' preinstalled and preconfigured. Let's call this Container B.
3. There should be Toxiproxy running on a different container. This container's image should be as light as possible and should have Toxiproxy installed. Let's call this Container C.

Container C will act as an intermediary for all network requests between Container A and Container B i.e. it will act like a man-in-the-middle proxy, which will enable us to introduce network failures between kqm and kafka.

The objective of the tests for this configuration is pretty straightforward:

1. Run kqm
2. Run the tests.
3. Introduce network faults between containers A and B from the tests (using Toxiproxy's http api).
4. Check kqm's behaviour (whether it fails or retries or something).
5. Remove the network faults, and run some normal tests. Check if kqm functions properly.
   By doing so we'll be able to test the fault handling capability of kqm w.r.t connecting to kafka.

.travis.yml needs to be setup to run the tests in the docker container. When this is done we should be able to simply have Travis integrated with Github, with it's status on the repository homepage.

The test configuration for Travis CI currently pull and tests the master branch only. This makes the running of tests for PRs on feature branches ineffective. Thus, we need to fetch the current branch from Travis CI's environment variables and pull the code from the concerned branch and run the tests accordingly.

Presently, consumeMessages() signals that the messages channel has been closed and that it's eligible for a retry (rather, a restart for this case). Theoretically, the message channel in not supposed to close on errors, since errors are separately written to another channel called errors (See: https://godoc.org/github.com/Shopify/sarama#PartitionConsumer).

We aren't using that errors channel and so we do not need to worry about the errors returned by sarama's PartitionConsumer. What's important for us is that if in the odds that the messages channel in the consumeMessages() closes, we need to restart it, so that KQM keeps functioning properly and is in sync with the __consumer_offsets topic.

Our aim for this issue is to try and minimize the complexity associated with the restart of consumeMessages().