aws.subscriber not thread safe about gizmo

[alessandrozucca]

One solution what we are thinking about is to create a pool of subscribers for a given queue, and each worker takes them from this pool

[jprobinson]

Howdy @alessandrozucca, when I'm fanning out messages to multiple workers from a single subscriber, I hand each goroutine a reference to the channel returned by Start(). That allows the subscriber to rely on the Go channel primitives to fan work out.

Is your problem that your bottleneck is consuming messages off of SNS? If that's the case, I'd suggest creating multiple subscriber instances and consuming them in separate goroutines/processes.

[jprobinson]

A very rough example of what I mentioned in my first block looks something like this:

func run(sub pubsub.Subscriber) {
	stream := sub.Start()

	for i := 0; i < maxWorkers; i++ {
		go consumeStream(stream)
	}

}

func consumeStream(stream <-chan pubsub.SubscriberMessage) {
	for item := range stream {
		fmt.Println(item)
	}
}

[alessandrozucca]

Thanks for your quick reply @jprobinson

Yes ideally we wanted to consume messages as fast as possible in parallel, but for now we will just pass in the channel to the workers.