kbs-asd / message-bus-performance-testing

KBS-ASD

message-bus-performance-testing's Issues

Cleanup BenchmarkContext

We can remove the TelemetryClient from the BenchmarkContext because it's only used in MessageCaptureContext.

Figure out what data is relevant when testing a message broker

Story: #24

[Faux-applications Send Messages] As tester, I want faux-applications that can send and receive messages through a message broker, so I can perform tests

Actors

• Tester
• Faux-application
• Message Broker

Prerequisite and/or assumptions

• Message broker is accessible

Acceptance criteria

• Must be scalable (so it is able to generate more messages when needed)
• Must be compatible with RabbitMQ and Azure Service Bus
• At least functional with in-memory
• Message frequency is random in small variations
• Message frequency is noise like at large

Description

• To run a test there has to be something to test on. In this case pretend applications (faux-applications) that will send and receive messages through MassTransit.

Exceptions

• Can not connect to message broker
  Faux-apps are stopped
  Error message is visible to the tester

Research how to measure the measurement points

[Faux-applications Send Messages] As tester, I want faux-applications which send messages through a message broker, so I can perform tests.

To run a test there has to be something to test on. In this case pretend applications (faux-applications) that will send and receive messages through MassTransit.

Actors

• Tester
• Faux-application
• Message Broker

Prerequisite and/or assumptions

• The application is automatically deployed
• A message broker is accessible

Acceptance criteria

• Must be scalable (so it is able to generate more messages when needed)
• Must be compatible with RabbitMQ and Azure Service Bus
• Message frequency is customizable by providing easing function
• Message size is customizable

Description
To run a test there has to be something to test on. We need an applications (faux-applications) that will send and receive messages through MassTransit.

Exceptions

• The request will fail when incorrect parameters are given

Additional information

This paragraph should contain any additional information like diagrams, images, datasets etc.

[Start Tests] As a tester, I want an application that can start a new tests with a custom configuration so I can easily and automatically collect reliable and meaningful information.

Actors

• Tester

Prerequisite and/or assumptions

• The tester wants to test a MassTransit configuration.

Acceptance criteria

• I can start a new test using a web interface
• I can customize the amount of messages
• I can customize the filler size
• I can customize the amount of clients
• I can customize the timeout duration
• I can customize the type of the testcase
• I can customize the type of the transport
• I can customize the timeout duration of the Azure service bus operation.

Description
As a tester you want to be able to start a test from an application. This application must connect to the TestSuite. By doing this the tester can easily start a test without having to do a lot of configurations, this makes testing faster and more reliable.

Exceptions

Add Rosylnator

Measure message broker performance by gathering telemetry data

Story #25

As a tester, I want the testsuite to be able to test all parameter combinations, so that I can give versatile test results.

Store test results on a place that is accessible by frontend

All measurements need to be quickly gathered and send allong for analysis on frontend. The json that is now flushed in InMemoryTelemetryClient should be saved on a place that is accessible by frontend.

Remove console readline from benchmark

[Gather test results] As a tester, I want to gather test results in MassTransit, so I have information to analyse performance.

Test results will be collected while the faux-apps are running. Telemetry data will be collected on certain points in the application flow, this data will be stored in a database.

Tasks

• Implement measurement tools to measure message broker performance #52
• Store test results #51

Actors

• Tester
• Faux-application
• Azure

Prerequisite and/or assumptions

• Application can send messages to a message broker
• Application can receive messages from a message broker

Post conditions

• Application has gathered test results that can be analysed
• Results are stored and ready to be analyzed

Acceptance criteria

• Should minimally impact application performance
• Should measure according to the details described in the research rapport
• Should store ready to analyse results for later use

Main scenario

1. Test wordt gestart in de faux application
2. The application gathers telemetry data during a test (1)
3. Results are stored

Exceptions

1. A test will be terminated when an error occurs during a test

Additional information

This paragraph should contain any additional information like diagrams, images, datasets etc.

Find suitable measurement points

Story #25.

We need to know what steps in the flow of sending a message with MassTransit are relevant.

Add Swagger for automatically generated api documentation

[Automatically create test environments] As a tester, I want to automatically create new tests, so there is no need for manual operations when testing a service bus

Actors

• KBS.Controller
• Azure App Service webjobs

Prerequisite and/or assumptions

• The application is automatically deployed with Azure DevOps
• The webjob that is used to run the benchmark is deployed

Acceptance criteria

• The test will run with the configuration that is passed through in the body of a POST request.
• Configuration in the body of the POST request is validated

Description
A webjob with the correct arguments is started when calling an api request on KBS.Controller. The webjob will do its thing and eventually save data in the background. The data can later be retrieved by the frontend where it can be analysed.

Exceptions

• Invalid configuration, there won't webjob shouldn't be triggered

Additional information

This paragraph should contain any additional information like diagrams, images, datasets etc.

Add c#/.net linting rules to .editorconfig

Add .NET coding convention settings to the .editorconfig.

Add EditorConfig

Rename methods in infrastructure manager

I think we should rename the methods in the infrastructure manager.

namespace KBS.Infrastructure
{
    public enum SortType
    {
        CreatedDate = 0,
    }

    public interface IManager
    {
        /// <summary>
        /// Get currently running tests
        /// </summary>
        /// <param name="sort"></param>
        /// <param name="isRunning"></param>
        TestEnvironment GetTests(SortType sort = SortType.CreatedDate, bool isRunning = false);

        /// <summary>
        /// Gets test status of test with given identifier
        /// </summary>
        /// <param name="identifier"></param>
        TestEnvironment GetTest(int identifier);

        /// <summary>
        /// Creates a test environment with the given configuration
        /// </summary>
        /// <param name="configuration"></param>
        void CreateTest(Configuration configuration);
    }
}

Create projects according to the package diagram

List of projects to create:

• KBS.MessageBus (ClassLibrary)
• KBS.MessageBusTests (XUnit)
• KBS.Messages (ClassLibrary)
• KBS.Controller (ASP .NET Application)
• KBS.ControllerTests (XUnit)
• KBS.Infrastructure (ClassLibrary)
• KBS.InfrastructureTests (XUnit)
• KBS.FauxApplication (ConsoleApplication)
• KBS.FauxApplicationTests (XUnit)

As a faux-application, I want to be able to follow all received parameters, so I know how I need to use the message broker.

Gather test data on specified measument points.

Story #24
Relates to #51

As a faux-application, I want to send messages to the message broker with MassTransit, so the testsuite can test the infrastructure.

As a tester, I want to collect telemetry, so I can analyse different test configurations.

[Configure service bus] As a tester, I want to configure the service bus for RabbitMQ and Azure Service Bus, so I can gather test results for these configurations.

[Receive parameters] As a faux-application, I want to be able to follow all received parameters, so I know how to use the message broker.

Actors

• Faux-application

Prerequisite and/or assumptions

• The application is automatically deployed with AppVeyor
• ...

Acceptance criteria

• Requirement 1
• ...

Description
This paragraph should contain a detailed description of this story

Exceptions

• The request will fail when incorrect parameters are given

Additional information

This paragraph should contain any additional information like diagrams, images, datasets etc.

Try Azure DevOps CI

As tester, I want faux-applications which send messages through a message broker, so I can perform tests.

1. Test Casus

[Receive Test Results] As a tester, I want to receive the test results through a HTTP GET request, so I can display the results in a dashboard.

Add CodeMaid config

[Test test suite parameters] As a tester, I want the test suite to be able to test all parameter combinations, so that I can give versatile test results.

Build cache is not working

The build cache is not working because packages are not cached in the packages directory when using .NET Core.

Update build status badge

[Analyse Data] As a tester, I want to be able to analyse data, so I can see which configurations and services are the best for my use case.

Actors

• Actor 1
• ...

Prerequisite and/or assumptions

• The application is automatically deployed with AppVeyor
• ...

Acceptance criteria

• Requirement 1
• ...

Description
This paragraph should contain a detailed description of this story

Exceptions

• The request will fail when incorrect parameters are given

Additional information

This paragraph should contain any additional information like diagrams, images, datasets etc.

#24 MassTransit has some statics

Describe the bug
The MassTransit framework uses statics classes and methods in the background. Meaning multithreading within the same application can't be easily done.

To Reproduce
Steps to reproduce the behavior:

1. Just run on any service bus

Expected behavior
Error log: "Another update request is in progress..."

Different Azure Service Bus implemenations

Is your feature request related to a problem? Please describe.
We only use the .NET Core version of Azure Service Bus. It would be interesting to see what the differences are between the MassTransit.Azure.ServiceBus.Core and other Azure service bus implementations.

• MassTransit.AzureServiceBus
• MassTransit.WebJobs.ServiceBus

Describe the solution you'd like
Create new transports for other Azure message bus implementations in KBS.MessageBus.Transports.

Create solution for project

Make initial project