universal-it-systems / p5-queue-q Goto Github PK

NAME
    Queue::Q - Mix-and-match Queue Implementations and Backends

SYNOPSIS
      # Pick you queue type and a back-end:
  
      # Very primitive FIFO queue (abstract interface)
      use Queue::Q::NaiveFIFO;
      # ... and its pure-Perl, in-memory implementation
      use Queue::Q::NaiveFIFO::Perl;
      # ... its Redis-based implementation
      use Queue::Q::NaiveFIFO::Redis;
  
      # "Reliable" enqueue, claim, mark-as-done FIFO queue:
      use Queue::Q::ClaimFIFO; # abstract interface
      # ... and again, its pure-Perl, in-memory implementation
      use Queue::Q::ClaimFIFO::Perl;
      # ... and its Redis-based implementation
      use Queue::Q::ClaimFIFO::Redis;
  
      # A composite, distributed queue, built from shards of
      # the above types with weak global and strict local
      # ordering, scaling beyond single nodes.
      use Queue::Q::DistFIFO; 
  
      # Usage is somewhat similar for most queue types:
      my $queue = Queue::Q::ClaimFIFO::Redis->new(
          server      => 'redis-01',
          port        => '6379',
          queue_name  => "image_queue"
      );
  
      $q->enqueue_item( $img_url );
  
      # In another process on another machine:
      while (defined( my $item = $q->claim_item )) {
          # Do some work!
          $q->mark_item_as_done($item);
      }

      # n-ary versions exist, too:
      $q->enqueue_items( @img_urls );
      my @items = $q->claim_items(10); # 10 at a time, padded with undef
      $q->mark_items_as_done(@items);

DESCRIPTION
    This is an experimental module. The interface may change without notice.
    Before using it in production, please get in touch with the authors!

    "Queue::Q" is a collection of queue implementations each with multiple
    backends. Right now, it comes with three basic queues:

    Queue::Q::NaiveFIFO
      A naive FIFO queue "NaiveFIFO". Supports enqueuing data and later
      claiming it in the order it was enqueued. Strict ordering,
      low-latency, high-throughput, no resilience against crashing workers.

    Queue::Q::ClaimFIFO
      A claim/mark-as-done FIFO queue "ClaimFIFO". Supports enqueuing data
      (as Queue::Q::ClaimFIFO::Item objects), claiming items, and keeping
      track of items-being-worked-on until they are reported as done. Strict
      ordering, slightly higher latency and lower throughput than the naive
      FIFO queue. Resilience against crashing workers when combined with a
      clean-up script that checks for old, claimed items.

    Queue::Q::ReliableFIFO
      Similar interface to the "ClaimFIFO" queue type, but only comes with a
      Redis-backed implementation now. Much faster for the basic operations
      then "ClaimFIFO" since it doesn't always have to execute Lua scripts.
      See Queue::Q::ReliableFIFO for details.

    The first two basic queues come with two back-end implementations each
    right now: One is a very simple, in-memory, single-process
    implementation Queue::Q::NaiveFIFO::Perl and Queue::Q::ClaimFIFO::Perl
    respectively. The other is an implementation based on Redis:
    Queue::Q::NaiveFIFO::Redis and Queue::Q::ClaimFIFO::Redis.

    As noted above, the "ReliableFIFO" queue has a Redis-based backend only
    for now.

    In addition to the basic queue types, the distribution contains
    Queue::Q::DistFIFO, an implementation of a distributed queue that can
    use the basic queues as shards (but only one type of shard per
    distributed queue). It supports all operations of the basic queues, but
    does not enforce strict global ordering, but weak global and strict
    local ordering. In an early test, two mid-range servers running multiple
    instances of Redis each sustained 800k-1M transactions per second with a
    naive-type distributed queue. "DistFIFO" has not been tested with the
    "ReliableFIFO" implementation as building blocks yet!

ACKNOWLEDGMENT
    This module was originally developed for Booking.com. With approval from
    Booking.com, this module was generalized and put on CPAN, for which the
    authors would like to express their gratitude.

AUTHOR
    Herald van der Breggen, <[email protected]>

    Steffen Mueller, <[email protected]>

COPYRIGHT AND LICENSE
    Copyright (C) 2012, 2013, 2014 by Steffen Mueller

    This library is free software; you can redistribute it and/or modify it
    under the same terms as Perl itself, either Perl version 5.8.1 or, at
    your option, any later version of Perl 5 you may have available.