beagle1984 / silverback Goto Github PK

Implement an IBroker to integrate with Azure Service Bus.

When I call the Execute/Publish method of a Publisher, this will lead into a construction of every ISubscriber, which is registered at the DI container.

This results in a huge overhead, as this will do a lot of unnecessary HeapAllocations, which again results in GC Pressure. As GC collects come at a cost, this should be reduced.

It would be nice to have support for a full component integration test using the in memory broker. Currently the serialization isn't tested.

A consumer scenario where byte[] is the input for the test:
message (byte[]) --> message (string) --> message (deserialized) --> ISubscriber (assert that we got the expected object)

A producer scenario where a object is the input for the test and the output is a byte[]:
IPublisher --> message (as object) --> message (deserialized) --> message (byte[]) (assert that we got the expected byte[])

I'd like to know if there is any configuration to force a message to be replicated in all brokers when I publish a message.
I had an issue with my application past week because one of my brokers went down and I lost the messages that were being published.
Do I need to change my KafkaProducerConfig in order to avoid it ?

Using Silverback.Integration.HealthChecks 1.0.4 and Microsoft.EntityFrameworkCore 3.0.1, the following exception is thrown executing the OutboundQueueHealthCheck.

[12:12:42 ERR - c8146ba0-4720943f5eccaf33] Health check OutboundQueue threw an unhandled exception after 189.6989ms <s:Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService>
System.InvalidOperationException: Processing of the LINQ expression 'DefaultIfEmpty<DateTime>(
    source: Select<OutboundMessage, DateTime>(
        source: Where<OutboundMessage>(
            source: DbSet<OutboundMessage>, 
            predicate: (m) => m.Produced == null), 
        selector: (m) => m.Created), 
    defaultValue: (Unhandled parameter: __p_0))' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core. See https://go.microsoft.com/fwlink/?
linkid=2101433 for more detailed information.
   at Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.ExpandAndReduce(Expression query, Boolean applyInclude)
   at Microsoft.EntityFrameworkCore.Query.Internal.NavigationExpandingExpressionVisitor.Expand(Expression query)
   at Microsoft.EntityFrameworkCore.Query.QueryTranslationPreprocessor.Process(Expression query)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken c
ancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.MinAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Silverback.Database.EfCoreQueryableExtensions.MinAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Silverback.Database.SilverbackQueryableExtensions.MinAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
   at Silverback.Messaging.Connectors.Repositories.DbOutboundQueueConsumer.GetMaxAge()
   at Silverback.Messaging.HealthChecks.OutboundQueueHealthCheckService.CheckIsHealthy(Nullable`1 maxAge, Nullable`1 maxQueueLength)
   at Silverback.Messaging.HealthChecks.OutboundQueueHealthCheck.CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken)
   at Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService.RunCheckAsync(IServiceScope scope, HealthCheckRegistration registration, CancellationToken cancellationToken)

Currently we need to get IHostApplicationLifetime & BusConfigurator in the Configure of the startup, to do the following.

IBroker broker = busConfigurator.Connect();
applicationLifetime.ApplicationStopping.Register(broker.Disconnect);

Can this be done behind the scene, that we can get rid of the two overloads and the two lines of code.

Hi! This lib is awesome, thank you ! ;)

Is there a way to support Avro in Kafka?

The DistributedBackgroundService sometimes throws an exception trying to insert the lock record into the database (violated PK constraint).

Hi Sergio

We currently have the situation where we decide how to deal with error conditions. Most likely we will cover a bunch of known scenarios with skips and retires. However, if something unknown happens, we will always fail i.e. stop the consumer.

Maybe there is an app with multiple consumers and just one is stopped. So bascially, the app still works fine. Until now we only see that a consumer was stopped, in the logs (of course we send alarms but thats more an event than a state). And the only thing we can to to reenable the consumer is to restart the whole app i.e. container.

1. Is it possible to detect the stopped consumer (e.g. Asp.net core HealthCheck -> Degreeded)
2. I would like to offer a management interface. In the simplest case, the consumer can be restarted. In more complex cases, this can be combined with "start, skip this message and continue".
   Thinking this further would lead to a API where you can ask for consumer status and start/stop a consumer.

Wdyt? Is it even possible at all? Is it compatible with the framework? What are the options?

I need to know from which partition a consumed message is from. Is it somehow possible to get this information on ISubscriber level (e.g. enriching after deserialization)?

If not, it would also be possible to maintain a global list which tracks, which Ids are from which partition. Is there a hook which allows to do so?

It is possible to have a unique topic to store more than one event type (type A and type B for example)?
If so, can I create multiple subscribers to manage it, one for the type A and other for type B to the same topic?

Hi Sergio

I've the situation were there is an application which has two consumers for two topics. Due to a authorization failure they cannot connect. I know that neither of both consumers can connect. However, this can not be inferred by looking at the log message.

  | 2020-01-08 18:58:19.910 loglevel="Information" sourceContext="Silverback.Messaging.Broker.KafkaBroker"  message="Creating new consumer for endpoint Topic1." exception="" {}
-- | --

  | 2020-01-08 18:58:19.932 loglevel="Information" sourceContext="Silverback.Messaging.Broker.KafkaBroker"  message="Creating new consumer for endpoint Topic2." exception="" {}
  | 2020-01-08 18:58:20.108 loglevel="Information" sourceContext="Post.SendingDataSystem.Service.Startup"  message="Application started." exception="" {}

  | 2020-01-08 18:58:21.034 loglevel="Warning" sourceContext="Silverback.Messaging.Broker.KafkaConsumer"  message="KafkaException occurred. The consumer will try to recover." exception="Confluent.Kafka.ConsumeException: Broker: Group authorization failed
  | at Confluent.Kafka.Consumer`2.ConsumeImpl[K,V](Int32 millisecondsTimeout, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
  | at Confluent.Kafka.Consumer`2.Consume(CancellationToken cancellationToken)
  | at Silverback.Messaging.Broker.InnerConsumerWrapper.ReceiveMessage()
  | at Silverback.Messaging.Broker.InnerConsumerWrapper.Consume()
  | " {}

  | 2020-01-08 18:58:21.033 loglevel="Warning" sourceContext="Silverback.Messaging.Broker.KafkaConsumer"  message="KafkaException occurred. The consumer will try to recover." exception="Confluent.Kafka.ConsumeException: Broker: Group authorization failed
  | at Confluent.Kafka.Consumer`2.ConsumeImpl[K,V](Int32 millisecondsTimeout, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
  | at Confluent.Kafka.Consumer`2.Consume(CancellationToken cancellationToken)
  | at Silverback.Messaging.Broker.InnerConsumerWrapper.ReceiveMessage()
  | at Silverback.Messaging.Broker.InnerConsumerWrapper.Consume()
  | " {}

Is it possible to include this information in the log statement?

I've the an contract first situation where we generate Poco types and distribute them as NuGet. The consumer as well as the producer relay on this NuGet. Thus, the Poco classes does not implement a Silverback specific interface nor inherit a type. This means, I've to register them using HandleMessagesOfType<>.

Therefore, I need to define a serializer and additionally, I also need to register the type. I would have expected that a type "on the bus" will just get routed to any subscriber which expects this type as parameter in a method.

Can you explain why it is necessary to explicitly register those types?

I configured chunking as described in the documentation, but while generating the DB migration I received the following error:

System.InvalidOperationException: Entity type 'TemporaryMessageChunk' has composite primary key defined with data annotations. To set composite primary key, use fluent API. 

To fix it in my EnityTypeConfigurator of 'TemporaryMessageChunk' I added the index configuration:

builder.HasKey(temporaryMessageChunk => new 
{ 
    temporaryMessageChunk.OriginalMessageId, 
    temporaryMessageChunk.ChunkId
});

Is this the right way?

My configuration looks like this.

  "Silverback": {
    "Using": [ "Silverback.Integration.Kafka", "Silverback.Core.Model", "**" ],
    "Inbound": [
      {
        "Endpoint": {
          "Type": "KafkaConsumerEndpoint",
          "Serializer": {
            "Type": "*RecordMessageSerializer"
          },
          "ErrorPolicies": [
            {
              "Type": "Retry",
              "MaxFailedAttempts": 5,
              "DelayIncrement": "00:05:00",
              "ApplyTo": [ "System.InvalidOperationException" ]
            }
          ],
          "Name": "*",
          "Configuration": {
                *
          }
        }
      }
    ],

The serializer:

    public class MailpieceRecordMessageSerializer : IMessageSerializer
    {
        public byte[] Serialize(object message)
        {
            throw new NotImplementedException();
        }

        public object Deserialize(byte[] message)
        {
            throw new InvalidOperationException("test");
        }
    }

[07:27:03 WRN] (Silverback.Messaging.ErrorHandling.InboundMessageProcessor) Error occurred processing the inbound message. {offset=*[0]@11, endpoint=*} System.InvalidOperationException: test

[07:27:03 FTL] (Silverback.Messaging.Broker.KafkaConsumer) Fatal error occurred consuming the message: *[0]@11. The consumer will be stopped. System.InvalidOperationException: test

Hi Sergio

See https://beagle1984.github.io/silverback/docs/configuration/external
This part raised an exception because the IProducer should be resolved. IProducer is scoped and there is no scope.

"ErrorPolicies": [
{
"Type": "Retry",
"MaxFailedAttempts": 5,
"DelayIncrement": "00:00:30"
},
{
"Type": "Move",
"Endpoint": {
"Type": "KafkaProducerEndpoint",
"Name": "basket-failedmessages"
}
}
]

With the newest version, AddXySubscriber has to be called on the ISilverbackBuilder. If I have correctly checked the code, this isn't necessary and those methods could also be extension methods of IServiceCollection.

As I have different modules which are responsible to register their components, this forces me to pass the ISilverbackBuilder.

See e.g. https://github.com/baernhaeckt/Backend/blob/master/Backend.Core/Features/Awards/Registrar.cs

What is the reason for the decision to offer this extension method for ISilverbackBuilder?

The Newtonsoft.Json library could be replaced by the brand new and faster System.Text.Json.

Hi Sergio

I thinking about how to handle errors and I've some questions.

Let me start how I think about exception handling, to give a context.
Basically, I want to have one global error handling in place, which ensures that all exceptions logged. With ASP.NET Core this is a middleware (shipped with the framework). Once I use the middleware, it is not necessary to catch exception which I can not handle.

I would like to have such a behavior also for my messages.
Basically there are three different "places" in the pipeline where errors can happen.

1. Within Silverback (e.g. Silverback can not parse a message with the default serializer).
2. Within Silverback-Extension points (e.g. my serializer or behavior throws an exception).
3. In my code, which is invoked as result of inbound message (e.g. a subscriber).

Different questions:

• What is logged by default in each of those three cases?
• Does every exception in those cases stop the consumer?
• Is it possible to enable the stopped consumer in code?
• Does a policy get applied for each of those three cases (especially before serialization)?
• What does the skip policy log?

Provide an example or an out-of-the-box IMessageSerializer that can encrypt and decrypt the messages being produced or consumed. An header should be added, indicating that the message is encrypted.

The JsonMessageSerializer has "DefaultValueHandling" and "NullValueHandling" set to ignore. This is unfortunate because it is not possible to distinguish if a Nullable was null or had the default value.

E.g. public bool? MyFlag; can be true, false and null but false and null will not be serialized.

Whats the idea behind the DefaultValueHandling?

The InMemoryBroker is causing a StackOverflowException when it gets disposed, causing the integration tests to fail.

(Thank you @gitjosephbeer for notifying it...the fix will be up in no time 😉)

At the moment the KafkaStatisticsEvent contains only the raw JSON as string.
It would be useful to convert that JSON into a strongly typed Dto.

I already created the necessary C# classes with correct Attributes for the JSON field mapping in one of my projects. I could share this work with you if you want.

im using the header x-message-id in a property in my code and its not being reflected in my event header when I publish it.

public class TurnoverData
{
        public Turnover Turnover { get; set; }
        public string AccountIdToBeActivated { get; set; }
}

public class Turnover
{
        [Header("x-message-id")]
        public string TurnoverId { get; set; }
        public string CreationDateTime { get; set; }
        public string PostingDate { get; set; }
}

I check the source code and found that just if the property is called Id or MessageId the lib retrieve the value and put it in the headers

In some situation you may want to subscribe to a topic with multiple consumer groups, to perform different parallel and independent tasks. The only way to do this at the moment is to subscribe to the IInboundMessage<T> and perform different operations according to the value of the Endpoint.Configuration.GroupId property.

It would be nice to have a cleaner way to filter incoming messages. For example using an attribute.

[KafkaGroupId("my-group-1")]
public void OnOrderReceived1(OrderCreatedEvent @event)
{
}

[KafkaGroupId("my-group-2")]
public void OnOrderReceived2(OrderCreatedEvent @event)
{
}

The filters attribute mechanism can be generic enough, to allow the creation of other filters in the future.

E.g. the following log line is a result of an execution outside of the activity-scope. Thus, the trace_id etc is missing.

2020-04-12 12:13:19.379 loglevel="Fatal" message="Fatal error occurred consuming the message 4@4071134 from endpoint logistics.parcel.staging.Pde-sdr. The consumer will be stopped." { offset="4@4071134", endpointName="logistics.parcel.staging.Pde-sdr", SourceContext="Silverback.Messaging.Broker.KafkaConsumer" }

Polly can be used to specify resilience policies. E.g. it can be used together with the HttpClient. Maybe it's worth to have a look and enable the same for Silverback (maybe in favor of the own implementation)

https://github.com/App-vNext/Polly & https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory

It is sometimes useful/necessary to know the Kafka key of the message being consumed. That value should be added to the IInboundMessage (or the new IInboundEnvelope).

The key could maybe be added as header, to avoid introducing the Kafka specific property in the IInboundMessage interface.

It would be useful in some use cases to be able to route the outbound messages according to a certain custom logic, instead of having them routed according to the message type.

It may look like this:

public class Startup
{
    public void Configure(BusConfigurator busConfigurator)
    {
        busConfigurator
            .Connect(endpoints => endpoints
                .AddOutbound<NotificationMessage, CustomRouter>(
                    new KafkaProducerEndpoint("order-events")
                    {
                        Configuration = new KafkaProducerConfig
                        {
                            BootstrapServers = "PLAINTEXT://kafka:9092"
                        }
                    }));
    }
}

public class CustomRouter : IOutboundRouter<NotificationMessage>
{
    public IProducerEndpoint RouteMessage(NotificationMessage message)
    {
        switch (message.NotificationType)
        {
            case NotificationType.Email:
                return new KafkaProducerEndpoint("mail-notifications") { ... };
            case NotificationType.Sms:
                return new KafkaProducerEndpoint("sms-notifications") { ... };
        }
    }
}

I need to get notified if the mentioned events happen.

As far as I see there is currently no possibility to set those handlers.
https://github.com/BEagle1984/silverback/blob/develop/src/Silverback.Integration.Kafka/Messaging/Broker/InnerConsumerWrapper.cs

As this is a blocker for my current project, I will gladly submit a PR. What do you think are good ways to expose access those handlers?

What do you think about adding support to Source Link to enable debbuging Silverback?

https://github.com/dotnet/sourcelink

As discussed. Something like this:

.SetErrorHandler((consumer, error) =>
{
    // Ignore errors if not consuming anymore
    // (lidrdkafka randomly throws some "brokers are down"
    // while disconnecting)
    if (!ownerConsumer.IsConsuming) return;

    var e = new KafkaErrorEvent(error);
    try
    {
        CreateScopeAndPublishEvent(e);
    }
    catch (Exception exception)
    {
        // TODO: Log
    }

    if (!e.Handled)
    {
        _logger.Log(
            error.IsFatal ? LogLevel.Critical : LogLevel.Error,
            "Error in Kafka consumer: {error} (topic(s): {topics})",
            error,
            ownerConsumer.Endpoint.Names);
    }
}

Currently the configuration method WithConnectionToKafka adds an extra behavior to handle the Kafka key (used for partitioning etc.). That behavior isn't added if the broker is registered via the generic method WithConnectionTo<KafkaBroker>.
This is misleading and the generic method should be removed (or hidden), if it cannot have an equivalent behavior.

Both implementations of the ExactlyOnceInboundConnector fail to take into account the Kafka group id. This means that if two consumer groups are sharing the same db (usually because running within the same process) only one of them will actually consume the messages.

The solution is pretty straightforward: add the consumer group id to the InboundMessages and StoredOffsets table.

Warning: try to preserve the compatibility with the older versions.

As reported in the discussion of the issue #75 the Silverback.Integration.Configuration package currently doesn't handle generic types.
This is the case reported in the issue #75:

  "Silverback": {
    "Using": [
      "Silverback.Integration.Kafka",
      "Silverback.Core.Model",
      "Silverback.Integration.Kafka.SchemaRegistry",
      "Super.GlobalPlatform.Authorizer.Intregation.Contracts"
    ],
    "Outbound": [
      {
        "MessageType": "RequestReceivedEvent",
        "Endpoint": {
          "Type": "KafkaProducerEndpoint",
          "Name": "cd-authorizer-received-requests",
          "Configuration": {
            "BootstrapServers": "PLAINTEXT://localhost:9092",
            "ClientId": "cd-ms-authorizer"
          },
          "Serializer": {
            "Type": "AvroMessageSerializer`1",
            "SchemaRegistryConfig": {
              "Url": "http://localhost:8081/"
            },
            "AvroSerializerConfig": {
              "AutoRegisterSchemas": true
            }
          }
        }
      }
    ]
  }

      Application startup exception
Silverback.Messaging.Configuration.SilverbackConfigurationException: Error in Outbound configuration section. See inner exception for details.
 ---> System.ArgumentException: Cannot create an instance of Silverback.Messaging.Serialization.AvroMessageSerializer`1[TMessage] because Type.ContainsGenericParameters is true.

Add an IBroker implementation for RabbitMQ.

Today I need to consume multitenant services, eg. repository.

In my Subscriber I get IServiceProvider from DI and create a scope on every consumed message:

    public class ExampleConsumer : ISubscriber
    {
        private readonly IServiceProvider _serviceProvider;

        public CreateCardConsumer(IServiceProvider serviceProvider)
        {
            this._serviceProvider = serviceProvider;
        }

        public async Task OnReceived(IInboundEnvelope<Event> message)
        {
                using (var scope = _serviceProvider.CreateScope())
                {
                    var tracer = scope.ServiceProvider.GetRequiredService<ITracer>();
                    var tenantContextFactory = scope.ServiceProvider.GetService<ITenantContextFactory>();

                     span.Span.Log("Set Tenant");
                     var key = message.Headers.GetValue("key");
                     tenantContextFactory.SetTenantIndentification(key);
                    var repo = scope.ServiceProvider.GetService<IRepo>();
                    await repo.SaveAsync(message.Message);
                }
        }
}

Is there a way to inject a ISeriveScope on message delegate ?

The initialDelay parameter on retry policy is not recognized. The retry is performed every 100milliseconds.

That one can make use of powerful log-aggregation tools like Splunk, it is very helpful to get structured log messages.

Here is an example of a log-message which isn't structured:

If I want to get out the count, I've to use regex. Logging it like this:
_logger.LogInformation("Processing the batch of {count} inbound messages.", {envelopes.Count});
Will (if one uses a corresponding logging framework) write the value of count not only inside the message but as own property.

It gets even more useful if every log entry makes use of EventIds because now I can get the information of an specific log message. I.e. I want the 95-percentil of batche sizes.

I'm glad to invest few hours to switch all statements to structured logging, if you agree that it's desirable. Might we can think about the EventIds as well.

TODOs

• Make all log messages structured.
• Add EventIds for all log events.
• Create a central class which holds all EventIds.
• EventIds class into different classes (according the the NuGet-Package structure)
• Split EventIds class into different classes (according the the NuGet-Package structure)
• Adjust/improve method names and text of the log messages

The KafkaProducer gets an "Broker: Invalid message" error when trying to produce to a topic configured with cleanup.policy=compact.

A single string property is decorated with [PartitioningKeyMember].

I'm writing an OpenTracing integration to Silverback in this repo, I think it would be nice if this integration was bult-in.

Is Behaviors the best approach ?

Implement an IBroker to integrate with Azure Event Hub.

Sometimes the DbOffsetStore throws a DbUpdateConcurrencyException when committing.
(The error isn't even handled by the policies and the consumer stops.)
This happens after having upgraded to the version 2.1.1.

Fatal error occurred consuming the message 286783 from endpoint *******. The consumer will be stopped.
Microsoft.EntityFrameworkCore.DbUpdateConcurrencyException: Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data may have been modified or deleted since entities were loaded. See http: //go.microsoft.com/fwlink/?LinkId=527962 for information on understanding and handling optimistic concurrency exceptions.
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ThrowAggregateUpdateConcurrencyException(Int32 commandIndex, Int32 expectedRowsAffected, Int32 rowsAffected)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeResultSetWithPropagationAsync(Int32 commandIndex, RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.AffectedCountModificationCommandBatch.ConsumeAsync(RelationalDataReader reader, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.ReaderModificationCommandBatch.ExecuteAsync(IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(IEnumerable`1 commandBatches, IRelationalConnection connection, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(IList`1 entriesToSave, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChangesAsync(DbContext _, Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerExecutionStrategy.ExecuteAsync[TState,TResult](TState state, Func`4 operation, Func`4 verifySucceeded, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(Boolean acceptAllChangesOnSuccess, CancellationToken cancellationToken)
   at Silverback.EntityFrameworkCore.DbContextEventsPublisher.ExecuteSaveTransaction(Func`1 saveChanges, Boolean async)
   at Silverback.EntityFrameworkCore.DbContextEventsPublisher.ExecuteSaveTransaction(Func`1 saveChanges, Boolean async)
   at Silverback.Messaging.Connectors.Repositories.DbOffsetStore.Commit()
   at Silverback.Util.EnumerableForEachExtensions.ForEachAsync[T](IEnumerable`1 source, Func`2 action)
   at Silverback.Util.TaskExtensions.GetReturnValue(Task task)   at Silverback.Util.EnumerableSelectExtensions.<>c__DisplayClass2_0`2.<<SelectAsync>b__0>d.MoveNext()

It would be a nice to have feature to be able to consume the same Kafka topic multiple times from within the same process (using multiple consumer groups). At the moment it’s not really feasible because all subscribers would be triggered every time a message is consumed but either group.

It’s maybe possible to do this already, subscribing to the InboundMessage instead and manually filtering according to the endpoint properties, but a more convenient way would be welcomed.

The IBehavior is a good way to plug-in functionality but an extension point closer to the Producer and Consumer is needed as well. This requirement emerged during the implementation of the Activity support, that required the direct extension of the Prodcuer and Consumer classes.

It would be great to have the chance to implement a sort of IProduerBehavior/IConsumerBehavior.

Version 2.2.0
Looking at the db OutboundMessage table, I noticed that the custom headers (set using the annotation) of my messages were not serialized and therefore sent.
I fixed the problem by registering a default outbound worker and assigning it to my outbound endpoint.
I hope it is useful to fix the problem.

It would be great to be able to configure multiple brokers (different products / different implementations of IBroker) to be mapped via inbound/outbound connectors.

A required feature in conjunction with Kafka on my part is the possibility to reset a topic-offset to the very beginning (offset 0, and maybe additionally to a certain timestamp).

This would allow to 'replay' messages, regardless if they had been consumed before already (see https://medium.com/@psinghal04/replaying-kafka-messages-an-implementation-in-golang-e5c6867cf084).

Such a topic offset reset should be possible at initial topic assignement and as well after a revoke and reassignment (e.g. consumer scale up/down).

I could imagine, that the KafkaConsumerConfig.AutoOffsetReset feature could therefor be enhanced by a new option AutoOffsetReset.Beginning (or First/Zero/Bottom?).

And/or in the KafkaPartitionsAssignedEvent handler allow access to the consumers of the IBroker and then call Acknowledge(offsets).
IBroker already has a GetConsumer() method, but the Broker base implementation there always creates and adds a new one in any case. My Suggestion would be to either rename this one to AddConsumer() and provide a 'readonly' GetConsumer() method or change the GetConsumer() method to the similar as Broker.GetProducer(IEndpoint) method which internally uses a _producers dictionary and _producers.GetOrAdd().

When does the release 2.1.0-rc8 will be GA?
i'm looking to forward to use this feature!

Support the subscription to messages with an empty body (IInboundMessage with just headers and key).