pablocastilla / nservicebus.kafka Goto Github PK

I have a simple application set up to test NServiceBus with Kafka. Finally set up ZooKeeper and Kakfa correctly.

But when running my .NET application I'm getting an error about that it couldn't parse the JSON correctly. What gives? See screenshot.

http://imgur.com/a/wD50W

Hi @danielmarbach @janovesk!

I would like to know how do you change the transport in the acceptance tests and transport tests. How do you tell, for instance , in the rabbit transport to use the rabbit instead of msmq. I don't get it.

Could you forward this to someone in particular that know it?

Thanks

While investigating issue #4 I was wondering why a console window pops up when the acceptance tests are executed. It turns out it is the queue creator. As far as I understand you are using the batch files to create the topics necessary to run the acceptance tests. While that works it makes debugging hard. On my machine every time I try to attach the debugger to an acceptance test I get the following exception:

System.IO.IOException : The pipe has been ended.

   at NServiceBus.AcceptanceTesting.Support.ScenarioRunner.<Run>d__0.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.AcceptanceTesting\Support\ScenarioRunner.cs:line 66
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.AcceptanceTesting.ScenarioWithContext`1.<Run>d__2.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.AcceptanceTesting\ScenarioWithContext.cs:line 57
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at NServiceBus.AcceptanceTesting.ScenarioWithContext`1.<NServiceBus-AcceptanceTesting-Support-IScenarioWithEndpointBehavior<TContext>-Run>d__9.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.AcceptanceTesting\ScenarioWithContext.cs:line 0
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at 

For better debugging experience I did the following:

• Replace the queue creator with an empty queue creator
• Enable

# Auto create topic
auto.create.topics.enable=true

in server.properties and restarted the broker.

I think shelling out into cmd.exe is not really a sustainable solution but unfortunately I don't have a good proposal to fix it since I'm still lacking lot of context so I'll just leave this here.

I have successfully sent a message using NServiceBus configure with a Kafka Transport, but am unable to receive the message unless listening to Kafka outside of NServiceBus. The following code blows up in a console app:

`var endpointConfiguration = new EndpointConfiguration("Sales");

		var transport = endpointConfiguration.UseTransport<KafkaTransport>();
		transport.ConnectionString("127.0.0.1:9092");

		endpointConfiguration.UseSerialization<JsonSerializer>();
		endpointConfiguration.UsePersistence<InMemoryPersistence>();
		endpointConfiguration.SendFailedMessagesTo("error");
		endpointConfiguration.EnableInstallers();

		var endpointInstance = await Endpoint.Start(endpointConfiguration).ConfigureAwait(false);

		Console.WriteLine("Press Enter to exit...");
		Console.ReadLine();

`

The error indicated is: 2017-03-14 14:23:37.032 WARN NServiceBus.Transport.Kafka.Receiving.MessagePump System.ComponentModel.Win32Exception (0x80004005): The directory name is invalid
at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start()
at NServiceBus.Transport.Kafka.Receiving.QueueCreator.d__3.MoveNext()

I'm using NServiceBus 6.1.3 and the latest Kafka Transport library from NuGet.

It might be that I don't yet fully understand the usage of the kafka clients. When I was looking at

https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L206

I had some spider senses tingling. For me the code read like

Whenever a message is completed we loop through OffsetsReceived https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L194 and commit all those that can be completed, essentially batching offsets to complete together but only triggered due to processing of messages.

Wouldn't it be more safe to do the following:

• Only complete things that can be truly associated with the message received

or apply smart background batching similar to what we do in Azure Service Bus.

Particular/NServiceBus.AzureServiceBus#384

The consumer holder uses

https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L186

and specifies ExecuteSynchronously

https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L229

although it actually executes IO-bound work

https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L225

There are multiple problems due to that:

If you stick to that pattern you have to remove the ExecuteSynchronously since you are violating the TPL constraints that a continuation with that flag should only execute synchronous and really short lived operations as well as mark the lambda as async and properly await the commit. If you don't do that you'll never find out when the commit fails

but overall I think this code is unnecessarily complex since you could write the same code inside the Process method with normal async await pattern

https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L262

Hi @danielmarbach @janovesk!

I hope you can help me with this issue that is making me crazy.

The problem is that when I use acceptance tests with addmapping I always get (for instance):
"No handlers could be found for message type: NServiceBus.AcceptanceTests.Basic.When_sending_to_another_endpoint+MyMessage"

I don't know what I am doing wrong.

Do you know what could be the reason for this?

Thanks.

Hi @ danielmarbach !

From the begining I wanted the transport elastic. Services with the same name are subscribed to the same kafka topic sharing it. The problem is I think it is impossible to implement the callback feature in the sending, as the sender could not receive the message if it is scaled out because it can be processed by another endpoint.

For me it is ok and I suppose this is a normal scenario with this kind of transports.

How do you see it?

Hi @danielmarbach !

I have a problem and I am not able to fix it. It is related to defering messages.

When I defer any message I have an exception related to a header called "Timeout.Id" but I don't know how to generate it and I don't find it in other transports.

What am I missing?

Thanks and have a good weekend!

Exception when executing When_deferring_to_non_local.Message_should_be_received:

Exception thrown: 'System.Collections.Generic.KeyNotFoundException' in mscorlib.dll
error onMessage: System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at NServiceBus.DispatchTimeoutBehavior.d__1.MoveNext() in C:\BuildAgent\work\3206e2123f54fce4\src\NServiceBus.Core\DelayedDelivery\TimeoutManager\DispatchTimeoutBehavior.cs:line 20
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable.ConfiguredTaskAwaiter.GetResult()
at NServiceBus.Transport.Kafka.Receiving.MessagePump.d__23.MoveNext() in C:\Users\sesa221523\Source\Repos\NServiceBus.Kafka\src\NServiceBus.Kafka\Receiving\MessagePump.cs:line 232

Hi!

As I understand transports need the message headers before pumping the message up in the next part of the pipe. So I need to get them from the message. My problem is that kafka doesn't have any headerbag for it, I only have a byte[] payload.

Do I serialize and deserialize every message at transport level? I could use
https://github.com/rogeralsing/Wire <-- i think now is the fastest
And a short message schema like headers/payload

The problem is that we will have to deserialize the message twice, one in the transport and another one at serialization/deserialization time.

Do you know any better option? @danielmarbach @sfarmar

Thanks

With v7 just right around the corner are there any plans to update the transport to support it? (there should not be any dramatic changes on the transport seam)

Let let me know if you need any help