grandecola / bigqueue Goto Github PK

Furthermore, EnsureArena should avoid removing HEAD from memory

Just curious, why mmap the files? It seems like simply writing serially and reading serially also should work, no? Would also give higher control over flushing and buffering.

@darkcoderrises

https://github.com/basho/bitcask

@mangalaman93 please issue a new release and update benchmarks!

We should expose the flush function as part of the bigqueue interface. Additionally, we should not trust the OS periodic syncing, and instead, enable flushing periodically, with a timer or probably by amount of data change, with configuration parameters to choose the period.

We can try two difference approaches -

• In this approach, we keep the queue single threaded and acquire lock from outside
• In this approach, we acquire locks on all the subcomponents such as index, arena.

GC will delete files from disk that are not in use anymore. This will be done periodically or on demand.

Benchmarks currently single threaded. Now that bigqueue is thread safe, we should be able to utilize the multi core performance for a higher throughput. We can also profile the system and improve the performance.

Given that it is a queue, I think we at least need following pages in memory -

• the page that is currently written
• the page that is currently read

Further, we can keep more pages after the currently read page in memory as well for more performance.

Whenever we mmap a new page, before mmap, we need to ensure that we do not cross the provided threshold limit for memory usage. If so, we remove some pages from memory before proceeding.

Refactored and added DequeueAppend([]byte) ([]byte, error)
Please review #84

We should use a Config object to allow configuring BigQueue. We should allow a useful default value for each configuration as well as ensure that configurations are set correctly using possible checks around each parameter. We should allow creating bigqueue without the config object using the default values.

Here is the list of configuration parameters -

• Maximum size of Arena (check for a value at least as much as size of a OS page)
• Maximum memory used by BigQueue, allow an option for using minimum possible memory, or all available memory (check for a value of at least as much as 2 * Size of Arena)
• GC frequency (check for a positive value)

We will have to persist the configuration parameters so that we can read these parameters back across different invocation of same application.

I still think that we should keep the directory as an argument to NewBigQueue to ensure an explicit invocation of creating a queue using a path. Given that path is what defines a BigQueue, the expectation will be set properly in that if you lose the directory, you lose the queue

This is similar to a producer-consumer model, except that queue can never be full. We should potentially add an API such that instead of returning an error when queue is empty, we simply block the caller until an element is added and can be returned.

There's no documentation on the thread-safety of the API. Would be great to have some explicit statements there, describing the current state of things.

Add benchmark to find performance for each function in BigQueue. This will answer following questions -

• Performance of read and write when size of the buffer is increased (with and without flush)
• Performance for creating queues and closing them

I'm trying to improve the throughput of an app I'm building. Essentially, it currently uses Unix Domain Sockets to transfer messages between two processes. This is because they are well supported in many programming languages, and easy to use for request/reply.

But combining the two process into a single process I get 10x the TPS. So I know there is up-to 10x potential improvement.

My questions is, can mmap do this, give some constraints:

• Request/reply (like HTTP).
• Concurrent, multiple request/replies in flight.

Bigqueue. Setarenasize() default 128M

Bigqueue. Setmaxinmememarenas() defaults to 3

Theoretically, the maximum number of Enqueue data is 128 * 3, but in practice, I can test unlimited Enqueue data.

How can I limit the Enqueue data to 1024m?

This is along the lines of how Kafka works. Given that bigqueue is persistent, it makes a lot of sense to not delete the data after it is read once. Certainly, it could be configured to do so, but that shouldn't be the default choice. Instead, we should allow using offsets per client so that the data could be read from anywhere in the queue as needed.

Currently, we need to first copy the string (say) into an array of bytes. Then, we will copy this array into the Arena. The double copy can be avoided.

A lock that provides write, read and delete lock.

Write Lock

Only one producer can write to BigQueue. Write is acceptable as long as it is at the tail of the queue.

Read Lock

Multiple consumers can simultaneously read from BigQueue. Read is acceptable as long as it is within the boundary of head to tail.

Delete Lock

This is fine, reads may fail if deletes are done before reads in the same region.

This would ensure that the same bigueue is not operated on (written/read) by multiple processes/threads. https://linux.die.net/man/2/flock

Reference: https://www.usenix.org/conference/atc20/presentation/rebello
Resources: https://github.com/wiscadsl

https://blog.dgraph.io/post/alice/