yodobrin / csv2q

Python function - insert large scale messages to queue

csv2q

Python function - insert large scale messages to queue

Context

When a quick, cheap & robust solution is required to process large scale activities in parallel, one of the enablers for this would be a message queue. within the queue, the asingle activities would be represented as messages within a queue. each handling node would dequeue a message from the queue, process it, and in most cases would log the success activity in designated queue, and the failure within specific error queue.
The solution of leveraging Azure Storage Queue fits into this, having segnificant low costs.

Architecture

Function

The function is triggered by messages inserted to the request queue. it assume the message is constructed of the required information to process single csv file and insert each line as message to provided queue. the function would create the queue in case it does not exist.

Storage Blob

the storage blob, is the location in which the function would load the csv file from. the details of the storage are part of the message.

Requests Queue

the request queue is where all requests for process should arrive to. it is expected a message has the following attributes in the following order, with comma (,) [0] - name of the input queue to send messages to [1] - storage account name [2] - storage account key [3] - container name for csv files [4] - csv file to use

Input Q

all the lines form the loaded csv would be inserted as seperated messages to the queue

provided handler

The provided handler is simple way to utilize the function

Build & Deploy

Update settings (queue names etc) and then run this from a command line

• func azure functionapp publish --build-native-deps

csv2q's Issues

art-diag