mongodb - use tailable cursors instead of polling about qu

[jnunemaker]

Correct me if I'm wrong, but wouldn't that limit the number of workers to one per queue or cursor? Also, there would need to be some work involved to make sure you didn't lose any jobs if the cursor closed I'm thinking.

On Thursday, November 3, 2011 at 1:39 PM, Greg Weber wrote:

• [http://www.captaincodeman.com/2011/05/28/simple-service-bus-message-queue-mongodb/](C# demonstration code)
• [https://gist.github.com/665978](Ruby tailable cursor code).

---

Reply to this email directly or view it on GitHub:
#33

[gregwebs]

Yes, you will still need to modify members of the collection to mark them
as finished so you know how to recover. This could be done by unsetting a
created_at field. You probably do only want to have one tailing cursor, but
that process could broker the queue to any number of consumers.
.
On Thu, Nov 3, 2011 at 11:42 AM, John Nunemaker <
[email protected]>wrote:

Correct me if I'm wrong, but wouldn't that limit the number of workers to
one per queue or cursor? Also, there would need to be some work involved to
make sure you didn't lose any jobs if the cursor closed I'm thinking.

On Thursday, November 3, 2011 at 1:39 PM, Greg Weber wrote:

• [
  http://www.captaincodeman.com/2011/05/28/simple-service-bus-message-queue-mongodb/](C#demonstration code)
• [https://gist.github.com/665978](Ruby tailable cursor code).

---

Reply to this email directly or view it on GitHub:
#33

---

Reply to this email directly or view it on GitHub:
#33 (comment)

[bkeepers]

That's a really interesting idea. It would be great if you could use a tailable cursor with the findAndModify command.

Any idea how we could do this without creating contention between workers?

[gregwebs]

You still have to use findAndModify to flag a document as finished
processing. Using the broker technique means only the broker accesses the
database, so there is no database contention between workers and the broker
can also supervise workers.

Other articles on this approach:

• a benchmark demonstrating very good performance [1]
• others using findAndModify instead of RabbitMQ [2]
• a Lift chat app using tailable cursors [3]

[1] http://www.shtylman.com/archives/217
[2] http://blog.boxedice.com/2011/09/28/replacing-rabbitmq-with-mongodb/
[3] http://amoss.me/lift-chat-with-mongodb-tailable-cursors

On Thu, Nov 3, 2011 at 1:38 PM, Brandon Keepers <
[email protected]>wrote:

That's a really interesting idea. It would be great if you could use a
tailable cursor with the findAndModify command.

Any idea how we could do this without creating contention between workers?

---

Reply to this email directly or view it on GitHub:
#33 (comment)

[mattparlane]

Hi all... I thought I'd drop by to say I've logged a feature request to have findAndModify return tailable cursors:

https://jira.mongodb.org/browse/SERVER-11753

This would allow blocking reads instead of polling (via tailable cursors), plus the guarantee that only one consumer would get any given job (via findAndModify). Please vote for it if you're interested in this functionality.

[jnunemaker]

If anyone wants to tackle this feel free to create a new backend like tailable_mongo or something.