pharaun / hsnsq

Haskell NSQ protocol parser

hsnsq

Haskell NSQ protocol parser

hsnsq's Issues

Benchmark hsnsq and nsq performance

We want to identify what the performance curves are for nsq so that we can decide if we should use it as a way to stream assets to the client or not.

The basic idea here is to have the standard two dimension benchmark in which:

1. X axis is the send-rate (ie 1msg/sec, 10msg/sec, etc...)
2. Y axis is the message-size (ie 1kb, 10kb, etc...)

Then in each message upon sending we will record the following 3 items:

1. Sequal id (to identify dropped/lost messages)
2. Timestamp of send-time
3. Gibbish for the rest of the message (from a block of memory for dma/mmap speed)

Then upon reception we will record the time the message was received, and record the send-time and sequal id to a csv/logfile for further analysis.

Should be able to make a binary that provides us with a few options such as send rate, message size, and so on for the initial benchmark effort. Binary provided options:

1. hostname/ip to connect to for nsq
2. port to connect to for nsq
3. count of messages to send
4. send-rate in msg/sec
5. message size in kilobytes (min of 1-4kb)
6. RDY size (we are doing 1 for now but should be able to bump this up)

Implement a test suite.

We currently don't have anything beyond the example program included.

Should probably implement:

1. Quickcheck for parsing
2. HUnit for some of the more specific stuff
3. Probably start looking at how to cleanly test the I/O portion, maybe free monad/type classes for this.

Primitive protocol is done. Need to finish the clientlib part.

Basically opening this ticket to track the feature/approach for the client.

Not sure if we need two threads, may need for concurrent PUB/MPUB support but for now most of the send ops can probably be done immediately. But anyway should have one thread that recieves messages and deals with heartbeat and all of that configuration stuff, then it pumps it into a queue in which it can monitor the high-water mark and adjust the RDY level for the client.

Then this should allow the library user to decide how they want to deal with the messages while letting the client deal with rdy, heartbeats, and all background stuff.

references:
http://www.yesodweb.com/blog/2014/03/network-conduit-async
https://github.com/informatikr/hedis
http://chimera.labs.oreilly.com/books/1230000000929/pt01.html

Fix up the subscription stage of the clientlib.

Currently the clientlib pre-auth for a hardcoded subscription.

Need to fix it up so that its optional to subscript if you want to listen otherwise don't.

1. Make subscription optional
2. Pass more messages to the client (errors/okays) so that it can manually manage subscription for now.

Fix the hardcoded client version to pull from cabal.

http://stackoverflow.com/questions/2892586/how-can-my-haskell-program-or-library-find-its-version-number

Fix up any issue with hackage.

http://hackage.haskell.org/package/hsnsq-0.1.0.0

The library is finally on hackage, but its still in alpha quality stage, need to fix the following issues so far:

1. Version bounds
2. Adding additional exports so its not just Network.NSQ but also the children modules
3. Improve documentation for hackage.

Pass more messages to the client (errors/okays) so that it can manually manage subscription for now.

split hsnsq into library/executable

we want to have the benchmarking stuff separate.