Asynchronous queueing about lxhive

[leogaggl]

I agree that employing some message queuing pattern would be a good idea. What MQ service have you got in mind for this ? Just in internal home-cooked MQ ?

In any case the use of an interface allowing the use of an internal MQ service that canpotentially be swapped for a separate MQ service like SQS or RabbitMQ would be good for larger scale deployments but probably out of scope for the moment.

[RoboSparrow]

Quick shot: All you need to store inside the queue is the statementId (either part of the request body or created before) and then re-visit the statements via a scheduler and process them. This would allow us still to return a posted statement by ID instantly albeit it might not be included in some search queries yet.

"processing" I mean readonly, extracting some meta data and store them into some fast lookup collections and the document apis.

Not sure how this would work with attachments as we store only fileUrls by convention. Worst solution would be that we change statement records after storage (performance, spec compliance).

[sraka1]

@leogaggl there are several backend-agnostic queue libraries available for PHP. Using one of them would give us the flexibility to change the queue provider dynamically (obviously).

However, seeing as we already use MongoDB and as you can basically store any document in it, I'd say initially the best bet would probably be to simply use mongo to enqueue and dequeue jobs.

[RoboSparrow]

agree with @sraka, the simplier the better

in regards to attachments. Store a fileURL in the statement with a naming convention /<storage>/<statementId>/<array_index_of_submitted_attachement>

Binary data are processing and moving are async, until then we return for this fileUrl:

202 Accepted

The request has been accepted for processing, but the processing has not been completed. The request might or might not eventually be acted upon, as it might be disallowed when processing actually takes place.

source