liumorgan / libqueue

C library providing persistent, named data storage queues w/ CLI utils. Uses LevelDB.

libqueue

libqueue is a C library that provides persistent, named data queues for programs. Arbitrarily many queues can co-exist, given they have distinct names.

Arbitrary binary data can be stored. Queues are manipulated using a stack-based API:

• push() -- append data to the queue
• pop() -- pop data from the queue
• peek() -- retrieve data from a given stack index

The data itself is stored using leveldb in a compressed hash tree.

API Overview

/*
  Intransparent data type. Nothing to see here...
*/
struct Queue;

/*
  Data structure used to store data in, and retrieve data from the
  queue
*/
struct QueueData {
  void *v;
  size_t vlen;
};

/*
  Functions return LIBQUEUE_SUCCESS on success, LIBQUEUE_FAILURE on
  error or LIBQUEUE_MEM_ERROR as a special failure type for failed
  memory allocations.
*/

/*
  Open a named queue. Directory location is provided as string
  <id>. Queues are persistent and data stored previously will still be
  available through the name it was written to. Operations on a queue
  must conclude with a call to queue_close().
  This function ensures that ${path}/libqueue exist. If it doesn't
  and cannot be created OR not written to, this function fails.
*/
struct Queue *  queue_open(const char *path);


/*
  Same as queue_open but allow one to provde options for various settings internal 
  to leveldb. The vararg must be terminated by a NULL to indicate end of options

  options are to be provided as strings with some requiring args

  available options:
  "failIfMissing" 

  if the database doesn't already exist, the database will not be created and error 
  status is returned

  "paranoidChecks"

  various paranoid checks will be done with every operation 

  "writeBufferSize"

  must be followed by size_t value. this is the write buffer size of leveldb

  "blockSize" 

  must be followed by size_t value. this is the block size of leveldb

  "blockRestartInterval" 

  must be followed by size_t value. this is the block restart interval of leveldb

  "maxOpenFiles" 

  must be followed by int value. max open file handles used by leveldb

  "noCompess" 

  disables compression
 */
struct Queue *  queue_open_with_options(const char *path,..., NULL);


/*
  Push a data object onto the queue. struct QueueData must be filled
  appropriately: d->v is a void* pointer to the data, d->vlen holds the
  data size.

*/
int queue_push(struct Queue *q, struct QueueData *d);

/*
  Pop data from the queue. The library will store the data in the passed
  QueueData struct. d->v must be freed by the user.
*/
int queue_pop(struct Queue *q, struct QueueData *d);

/*
  returns approximate leveldb size if there is at least one element ready 
  for reading in queue, other wise returns 0

*/
int queue_len(struct Queue *q, int64_t *len);

/*
  returns numbers of elements in the queue
*/
int queue_count(struct Queue *q, int64_t *countp);

/*
  compacts the queue on file system to use take up less disk space if possible
*/
int queue_compact(struct Queue *q);
/*
  Like queue_pop(), but doesn't remove the data object from the queue
  and returns the data at queue position s. Indexing is not used 
*/
int queue_peek(struct Queue *q, int64_t s, struct QueueData *d);

/*
   Like queue_push(), but over-writes the data at queue position s. This
   function cannot be used to append new data to the queue. Indexing
   starts from zero. s must be positive.
 */
int queue_poke(struct Queue *q, int64_t s, struct QueueData *d);


/*
  Closes the queue. Only AFTER this function has been called one can be
  sure that the data has been properly stored. Undefined behaviour may
  occur if the concluding call to this function is omitted.
*/
int queue_close(struct Queue *q);

struct Queue is an opaque data type the fields of which should not be accessed directly.

Building

Dependencies:

• libleveldb (on ubuntu, apt get install libleveldb-dev)

libqueue uses the cmake for building:

cmake . make

Usage

It's very simple: Use

#include <queue.h>

in your C program and link it with

-lqueue -lleveldb

given that libqueue was installed into the default linker search path.

queueutils

The library comes with a number of reference/test/practical utilities that all operate on the queue named 'queueutils-stack0' to be used on the command line:

qpush

Pushes all its arguments in onto the queue queueutils-stack0. Exits with an error code of 0 if the push was successful and 1 if it wasn't.

qpop

Pops the strings from the queue that were pushed using qpush, on element per call to qpush. If there is no element to pop, exits with the status code 1. If the pop was successfu, it prints the string to stdout and exists with the status code 0.

qlen

if there are no elements, then 0 is returned, otherwise approximate length from leveldb is provided

qcount

number of elments in the queue

qcompact

compact the queue on the file system to use less diskspace

qrepair

repair a broken leveldb

qpeek

Peek at elements in the queueutils' queue.

qpeek <INDEX> [<index1>[..<indexN>]]</indexN>]]

Tries to peek at all indizes listed as arguments. Skips over invalid indizes but fails if the queue cannot be accessed properly.

qpoke

Replace an existing in the queue with another one.

qpoke Fails if we cannot poke the element at the specified index, or if the index is out of bounds.

qtest

executes some basic tests

License

This program is licensed under the GNU Lesser General Public License v3 or later. Note that the LGPL is a set of additional terms that apply on top of the GPL. The text of the GPL3 is provided in LICENSE.GPL3, the text of the LGPL in LICENSE.LGPL3.