The Nuget package here should include the .pdb files as well as the .dll files. I would put in a PR for this, but I can't find the .nuspec file for that nuget package. How is it built?

I would like to be able to know how long a message has spent on a queue before being successfully processed - so the difference between publish time and successfully-processed time.

I would like that stat to be possible to publish to my monitoring, so I can alert when the time-on-queue is too high.

PR ready: #26

Hi,

Is there a way to inject a message mutator for publisher? Interested for Sns publisher only for the moment.
I.e. we want to add extra stuff to the message before publishing. Such as timestamp, authentication token, user context, etc, or anything else that can be extracted to a separate message mutator.

I see there's only serialization happening in SnsTopicBase.Publish(Message), is there an extension point I could use to modify the message just before publishing?

Thanks

Currently, publish() is a synchronous call in a background thread. Which means potential errors will NOT be handled.

Full of JE specific stuff etc.

Sometimes we want to delay the the publishing of a notification to a topic for extended periods of time. SNS doesn't support this out of the box.

Example Scenarios when this could be used

1. When a user registers with my website, I send them a confirmation email with a confirmation link in the email. If the user doesn't respond in 7 days I want to notify the rest of the system that they haven't responded.so that a reminder email can be sent to the user. I would do this by publishing a notification with a delay of 7 days (i.e. it would not be published to subscribers until exactly 7 days had passed). If the user does click on the confirmation link within 7 days then I need to be able to delete the notification so it does not get published to subscribers.
2. On my shopping website, when a user adds items to their basket I need to reserve the stock for an hour. After an hour I need to be able to release the stock. I would do this by publishing a notification with a delay of 1 hour. If the user does not checkout within an hour then the notification is sent to subscribers so my reserved stock is released. If the user completes their order within an hour then I need to be able to delete the notification so it is never published to subscribers.

When I call bus.StartListening(), I can see that be logged twice.

I expect it to only be started once - or is this a case of logging being misworded?

[ExactlyOnce(Timeout=30)] will ensure the message can only be handled within the first 30 seconds.[ExactlyOnce] should ensure message can only be handled indefinitely but it doesn't work as expected because timeout defaults to 0.

For existing SQS queue with visibility timeout of 30 seconds.
The following declaration is expected to increase the visibility timeout to 60 seconds but that is currently not the case.

JustSaying.CreateMeABus.InRegion("eu-west-1")
    .WithSqsTopicSubscriber().IntoQueue("consumer")
    .ConfigureSubscriptionWith(cnf => cnf.VisibilityTimeoutSeconds = 60)
    .WithMessageHandler(handler);

We use AWS SWK version 2.3.x throughout, but the latest stable version of the AWS SDK is 3.1.x.

At some stage we will want to take on the breaking changes of major version number 3. One thing that will be important is DotNet Core support, which will land in version 3.2 which is currently in beta

Are there any other benefits of using version 3.x? I know it is much more modular, so there will be more packages but a smaller total size of binaries used. Are there any features that are not present in the version 2 packages that are useful?

I have done some experiments, and have got a somewhat working CI on appveyor.

You can see the badge icon in action on this branch, just near 'gitter' icon. There are a number of failing tests there, but this serves well as a proof of concept that this can be done.

1. Not sure what to do with failing DynamoDB tests, do we really need those here?
2. There are some other failing tests, will need to look at them.

Interested to hear your thoughts of this, whether this is something that would be of use.

All settings related to the configuration of the message have the term "Message" as a prefix, VisibilityTimeoutSeconds is an exception to this rule.

Suggest changing this to make it clear as to what the configuration applies to.

    CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsTopicSubscriber()
                .IntoQueue("CustomerOrders")
                .ConfigureSubscriptionWith(config =>
                {
                      config.MessageRetentionSeconds = 60;
                      config.VisibilityTimeoutSeconds = 30;
                }
                .WithMessageHandler<OrderAccepted>(new OrderNotifier())
                .StartListening();

would become

    CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsTopicSubscriber()
                .IntoQueue("CustomerOrders")
                .ConfigureSubscriptionWith(config =>
                {
                      config.MessageRetentionSeconds = 60;
                      config.MessageVisibilityTimeoutSeconds = 30;
                }
                .WithMessageHandler<OrderAccepted>(new OrderNotifier())
                .StartListening();

Currently, MessageId is always empty GUID.

Answer some common integration questions

If you have a handler that depends on transient disposable objects, like db connections or sessions etc, you need to get hold of the IoC container in your handler and manually resolve and release those dependencies.

This behaviour should be pushed into JustSaying as it is generic enough.
So we need to support something like:

..WithMessageHandler<OrderAccepted>(new HandlerResolver())

The hash used for guaranteed once delivery should include handler's type name as discriminator otherwise handlers can interfere with each other.

Hi,
The default way to override queue parameters is throught ConfigureSubscriptionWith as in.

CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
             .WithSqsTopicSubscriber()
             .IntoQueue(queue2)
             .ConfigureSubscriptionWith(y =>
             {
                 y.MaxAllowedMessagesInFlight = 1;
             })

Now, I wonder would it be a good idea to update JustSayingConstants and instead of constants change default parameters to properties. This is a minimal change, no impact, but would allow us to change the defaults for subscribers in a global way. We have a scenario where we have tens of services, and now they all have a default parameters set to what is defined in JustSayingConstants. Now, without going to each of the service and updating it manually, which obviously would be a tedious and repetitive job, we would like to just update this globally. All of our services use an abstraction that we have built around JustSaying, so definitely this change would allow us to achieve this.

Let me know what you guys think about it, happy to send a PR for this.

Hi,

I see there's IDynamoStore.cs file in AwsTools, is this something internal to what JustEat uses?

...so that i/o bound handlers become more resource-efficient.

Had a preliminary chat with @payman81 and decided to go for something along the lines of adding IAsyncHandler<T> interface, similarly to IHandler<T>.

This way, the change will be non-breaking and it will be up to users to pick whether they want to go async in a particular scenario.

In the Appveyor output we get this text

NUnit-Console version 2.6.4.14350
Copyright (C) 2002-2012 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment - 
   OS Version: Microsoft Windows NT 6.2.9200.0
  CLR Version: 2.0.50727.8009 ( Net 3.5 )

ProcessModel: Default    DomainUsage: Multiple
Execution Runtime: net-3.5

This is of course wrong, we are on .Net version 4.5.2.

We are experiencing this issue. This affects some async tests - we have had spurious fails on appveyor of tests that pass locally, and we likely have spurious passes (i.e. if they started failing locally, appveyor would not notice).

Using NUnit 3.0 may be the answer according so some discussions. We don't use NUnit 3.0 yet.
JustSaying my be a good trial for NUnit 3.0?

Hi

There's an issue with how SqsPointToPointSubscriber is creating queues. Below example should just create 1 queue, and should handle the message. Instead, it creates 3 queues and does not handle a message.

var queue = "MyQueue";
var publisher = CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsPointToPointSubscriber()
                .IntoQueue(queue)
                .WithMessageHandler<Message1>(new Handler1())
                .WithMessageHandler<Message2>(new Handler2())
                .WithSqsMessagePublisher<Message1>(config => config.QueueName = queue)
                .WithSqsMessagePublisher<Message2>(config => config.QueueName = queue);

publisher.StartListening();

publisher.Publish(new Message2());

Code creates 3 queues:

• message1-myqueue
• message2-myqueue
• message2-message1-myqueue

SQS subscriber is listening on message1-myqueue and messagee2-message1-myqueue
Message is published on queue message2-myqueue

I see a couple of issues with this behaviour

1. The code does add a prefix to the queue name which is the message name. This is not evident from the fluent interface. I believe the intention here was to create queue per message, so that the bus would listen on each queue. This is also different behaviour compared to SnsSubscriber, where only a single queue will be created. Perhaps there's a bug in there as well - maybe the intention is to have a queue per message type, and not a single queue with multiple listeners as it is now.
2. Ref what has been said above, I expect this code to create just 1 queue. And not 3.
3. Message is never handled due to a mismatch in queues

I would like to have a way of knowing why a message was processed - where it originated from, which user at the application-layer caused that message to be published and consumed, and things like that. I tend to think of messages like deferred HTTP requests.

In the HTTP world, I'd be able to add headers to the request like x-conversation with some unique ID, that can then be harvested & passed through as work ripples around a distributed system. This is really useful for tracing work and user journeys. I'm able to use the same mechanism to decide how to handle requests, for example if I'm A/B testing.

Having talked to @payman81, we think this is possible via something like

• JustSaying makes a message Envelope (analogous to an HTTP request body) a 1st-class citizen, and available to handlers when they process messages.
  • the envelope can basically expose the SQS metadata, which opens up a set of enhancements like being able to know dwell times of the underlying transport, how many attempts that the message has had by a given handler, and some other stuff
• To (try to) avoid a breaking change, we talked about introducing a base class for Handler with a virtual method that consumers can choose to override, or not, which has access to the Envelope, and can make decisions based on it.

Hi,

Currently JustSaying posts messages which are in this format

{ 
     Subject = <messageType>,
     Message = <messageBody>
}

This works well when using JustSaying to send and receive messages. However, this is also a limitation because there are cases when we are not in control of the incoming message format.

One example is using SES to send emails. This service also allows to listen for delivery notifications, such as bounce notifications. In this case, SES publishes notifications to our topic of choice. When JustSaying polls for new messages, the received message looks completely different:

{
  "Type" : "Notification",
  "MessageId" : "--id--",
  "TopicArn" : "arn:aws:sns:eu-west-1:963735208092:--topicname--",
  "Message" : "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceSubType\":\"General\",\"bounceType\":\"Permanent\",\"reportingMTA\":\"dsn; a6-180.smtp-out.eu-west-1.amazonses.com\",\"bouncedRecipients\":[{\"emailAddress\":\"[email protected]\",\"status\":\"5.1.1\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\",\"action\":\"failed\"}],\"timestamp\":\"2015-06-05T13:48:57.172Z\",\"feedbackId\":\"0000014dc3fb5186-78d7691e-7665-47f9-8f81-182e8da28ba1-000000\"},\"mail\":{\"timestamp\":\"2015-06-05T13:48:55.000Z\",\"source\":\"--sourceemail--\",\"messageId\":\"0000014dc3fb4cac-23402d11-bfe7-4735-b663-5b6eca2f0daf-000000\",\"destination\":[\"[email protected]\",\"--dest-email--\"]}}",  
  "Timestamp" : "2015-06-05T13:48:57.223Z",
  "SignatureVersion" : "1",
  "Signature" : "--..",
  "SigningCertURL" : "--..--",
  "UnsubscribeURL" : "--..--"
}

What we'd like to do is to be able to add custom serializators, which would be able to parse this message. IMessageSerialisationFactory is already acting in that area, but what will need to be changed is how the message type is discovered (currently that is built-in), and where the message body is being loaded from.

We are working on this feature, will give update once we have something in this area.

Hi,

Any reasons against upgrading NLog to 4.x ?

We don't need it in there.
Default to Newtonsoft & allow an override

Hi,

This is more of a question not an issue. It's note exactly clear from docs whether we can have handlers for more than one message. I assume it is, so this is what it looks like:

CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
             .WithSqsTopicSubscriber()
             .IntoQueue(queue)
             .WithMessageHandler<SubscriptionAdded>(new SubscriptionHandler())
             .WithMessageHandler<SubscriptionRemoved>(new SubscriptionHandler())
             .StartListening();

Now, looking at JustSaying.Bus.cs:86, the Start() method would have 2 subcsribers, because there are 2 subscribers by topic. However, both topic subscribers are listening for the same queue, so is it the case that both listeners will poll the same queue at same time?

This all makes sense for multiple regions - needs to listen for each region separately as been noted in issue 59. But this case is just for single region single queue.

I might have completely misunderstood how configuration is supposed to be used for multiple handlers.

Thanks

This is because the value for retention period of error queue is replaced by retention period of the actual queue.

The collection Subscribers on SqsNotificationListener (SqsNotificationListener.cs#L219) is only added to by its own AddMessageHandler method (SqsNotificationListener.cs#L86). JustSayingFluently has its own AddMessageHandler methods that create an SqsNotificationListener per Subscriber (JustSayingFluently.cs#L283). Seems like the class SqsNotificationListener is incorrectly serving 2 different purposes here.

Therefore subscribers are not added to the interrogation response when using JustSayingFluently (JustSayingBus.cs#L82).

Not all of our services need to subscribe to bus messages, some of them just need to publish.

Below code does not compile, unless WithMessageHandler is uncommented.

justSaying
    .WithSqsTopicSubscriber()
    .IntoQueue(busNamingConvention.QueueName())
    //.WithMessageHandler<SimpleBusMessage>(handlerResolver)
    .WithSqsMessagePublisher<UnscheduleTimeout>(c => c.QueueName = "<queueName>"));

We have:

• JustSaying.Models.sln
• JustSaying.sln

When building in AppVeyor using appveyor.yml (i.e. not using the settings in the UI), we see a complaint about having multiple sln file. This is despite the yml contaning project: JustSaying.sln.

We should just stick the Models project into the main solution. Having them separated out has give little/no value.

Setup wiki pages.

• Common set-up scenarios
• Anything else......

i.e. in https://github.com/justeat/JustSaying/blob/master/JustSaying.AwsTools/SqsNotificationListener.cs#L258

If you receive a message, you must be able to deserialise it.

We want to be able to test on whether messages where published to correct queue or topic.

In its current state, JustSaying does not let us control how SnsTopicBase or SqsPublisher are created, and thus we can not add any custom logging that we need.

Suggestion is to provide hooks to control how sns topic bace and sqs publisher are created.

Is is advisable to increase the .Net build version to 4.5.0?
Will this adversely impact any clients?
This would be required before doing any async handlers, but can be done as a separate change.

You can currently specify the "MaxAllowedMessagesInFlight" in two ways which conflict.

var throttlingStrategy = new Throttled(() => 10, 10, messageMonitor);

 CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsTopicSubscriber()
                .IntoQueue("CustomerOrders")
                .ConfigureSubscriptionWith(config =>
                {
                      config .MaxAllowedMessagesInFlight = 10;
                      config.MessageProcessingStrategy = throttlingStrategy;
                }
                .WithMessageHandler<OrderAccepted>(new OrderNotifier())
                .StartListening();

Suggest removing removing one of these options. I personally prefer removing it from the ConfigureSubscriptionWith as the context of this option is related to throttling.

var throttlingStrategy = new Throttled(() => 10, 10, messageMonitor);

 CreateMeABus.InRegion(RegionEndpoint.EUWest1.SystemName)
                .WithSqsTopicSubscriber()
                .IntoQueue("CustomerOrders")
                .ConfigureSubscriptionWith(config =>
                {
                      config.MessageProcessingStrategy = throttlingStrategy;
                }
                .WithMessageHandler<OrderAccepted>(new OrderNotifier())
                .StartListening();

Hi,

What is the best way to provide a different IPublishEndpointProvider implementation when registering publishers and subscribing to topics?

Can't find how to do that, all I can see is that JustSayingFluently.cs:70 calls CreatePublisherEndpointProvider(..) which creates news a new instance.

Ultimately i want to configure topic name, i.e. have it configurable based on a number of environment params, not only just message type.

A 'bus' has these 3 components

1. A Topic
2. A Queue
3. A Subscription between Topic and Queue

Responsibilities

AccountA
Owns Topic
Applies policy to allow AccountB to subscribe

AccountB
Owns Queue
Owns Subscription
Must know the ARN of the the AccountA Topic (in order to create subscription)
Applies policy to queue to allow AccountA Topic to deliver messages

When handlers take longer than 15 seconds the temporary lock against the message will be removed and message will be handled again. The desirable case is for message lock timeout to be customisable for each handler.

[ExactlyOnce(Timeout=60)]
public class MyHandler : IHandler{...}

Hi,

For each topic handler that we register, there is a call to ensure that topic and queues are created. I really love how you can just fire JustSaying, and it will auto create topics and queues for you. There is some inefficiency in that though.

In AmazonQueueCreator.cs:35 a SqsQueueByName is created. Inside constructor it calls Exists() to check whether that queue exists. Then again on the next line immediately afterwards, in call to EnsureQueueAndErrorQueueExistAndAllAttributesAreUpdated again the call is made to Exists() which again queries SQS and lists the queus.

So it's checking whether a queue exists 2 times in a row. Furthermore, it will perform those same checks against the same queue when the next topic handler will be registered. So we have here 2*M number of checks whether a single queue exists, where M is the number of topic handlers for given queue. It is possible to update this to just do a single call instead.

foo!

These tests give us greater confidence in the changes we make. There is an issue in that they require a functioning AWS environment to execute.

Hi,

Messages are serialized differently when using SQS publisher instead of SNS publisher. SnsTopicBase finds a type serializer and uses that to serialize message. However, SqsPublisher is using JsonConvert.SerializeObject to serialize message, and not message serializer.

Not really an Issue. I just wanted to start a dialog about what we can do on the AWS .NET SDK side to help out.

As someone new to JustSaying I tried following the tutorial to set up a new component that listens to messages from a given topic.

The tutorial doesnt touch upon credentials or where to use acces and secret key

In all the JustSaying.AwsTools.UnitTests.MessageHandling.SqsNotificationListener.WhenMessageHandlingXXX tests, I can change Monitor.ReceivedWithAnyArgs().HandleException to Monitor.DidNotReceiveWithAnyArgs().HandleException and vice versa and the test still passes.

This means that they are not actually testing anything. It is likely that the async has gone wrong, something is declaring success too early, instead of awaiting actual pass or fail.

I have also had issues on a PR with tests that pass locally and fail on Appveyor, after emitting so much output that the Appveyor build results cannot be viewed.

The combination of JustBehave and await Patiently.VerifyExpectationAsync seems to be a bad one.

Just forked the repo, ran tests and couple of them are failing out of the box. Ran the entire test suite a few times and it seems some tests fail due to timing-related issue, though there seems to be some repeat offenders:

Need to be able to verify what SNS/SQS messages we are publishing.

Several ways to do that:

• Mock IAmazonSimpleNotificationService, (and SQS counterpart) and verify on Publish method
• Create custom AmazonSimpleNotificationServiceConfig, and set ServiceURL property. Then use this along with WireMock

In any case, factory for creating SNS/SQS clients should be exposed.