GithubHelp home page GithubHelp logo

testenvironment-docker's Introduction

Build status NuGet

Test environment with Docker containers

Pre requirements

You need docker to be installed on your machine. Tested both on Windows and Linux.

Installation

Run this in your package manager console:

 Install-Package TestEnvironment.Docker

To add container specific functionality for MSSQL, Elasticsearch or MongoDB:

 Install-Package TestEnvironment.Docker.Containers.Elasticsearch
 Install-Package TestEnvironment.Docker.Containers.Mssql
 Install-Package TestEnvironment.Docker.Containers.Mongo
 Install-Package TestEnvironment.Docker.Containers.Mail
 Install-Package TestEnvironment.Docker.Containers.Ftp
 Install-Package TestEnvironment.Docker.Containers.MariaDB
 Install-Package TestEnvironment.Docker.Containers.Postgres
 Install-Package TestEnvironment.Docker.Containers.Kafka
 Install-Package TestEnvironment.Docker.Containers.Redis
 Install-Package TestEnvironment.Docker.Containers.RabbitMQ

Example

Latest version is heavily using C# 9 records feature. Please make sure you are targeting net5.0 and above.

// Create the environment using builder pattern.
await using var environment =  new DockerEnvironmentBuilder()
    .AddContainer(p => p with
    {
        Name = "my-nginx",
        ImageName = "nginx"
    })
    .AddElasticsearchContainer(p => p with
    {
        Name = "my-elastic"
    })
    .AddMssqlContainer(p => p with
    {
        Name = "my-mssql",
        SAPassword = "HelloK11tt_0"
    })
    .AddPostgresContainer(p => p with
    {
        Name = "my-postgres"
    })
    .AddFromDockerfile(p => p with
    {
        Name = "from-file",
        Dockerfile = "Dockerfile",
        ContainerWaiter = new HttpContainerWaiter("/", port: 8080)
    })
    .Build();

// Up it.
await environment.UpAsync();

// Play with containers.
var mssql = environment.GetContainer<MssqlContainer>("my-mssql");
var elastic = environment.GetContainer<ElasticsearchContainer>("my-elastic");
var postgres = environment.GetContainer<PostgresContainer>("my-postgres");

Docker Desktop subscription changes

As you may know, Docker announced subscription changes. So you can not use Docker Desktop for free anymore (except for personal purposes). Good news is that Docker for Linux is still free (including Docker Engine, Docker Daemon, Docker CLI, Docker Compose, BuildKit, libraries, etc). So we have several options:

Option 1 - Run tests on Linux/WSL2

You can setup WSL2 on your Windows machine, install .NET and Docker (also, full instruction for Docker setup could be found here). Then just navigate to your source code and simply run:

dotnet test

Option 2 - Expose Docker Daemon from WSL2 to Windows and proceed using Visual Studio

Ok, you still want to use Visual Studio to run and debug your tests. So this is still possible. First, you need to expose your Docker Daemon installed to WSL2. For this, go to /etc/docker/ and create/edit daemon.js file:

{
  "hosts": ["tcp://0.0.0.0:2375", "unix:///var/run/docker.sock"]
}

Then add UseWsl2() option to your tests:

await using var environment =  new DockerEnvironmentBuilder()
    .UseWsl2()
    .AddMssqlContainer(p => p with
    {
        Name = "my-mssql",
        SAPassword = "HelloK11tt_0"
    })

And that's it!

Official instruction regarding Docker remote access are here.

Podman support

As a container engine you can use Podman instead of docker. It works the same and even has the same api. To install podman on Windows and use it follow this instruction (Super easy).

Troubleshooting

In case of unpredictable behaviour try to remove the containers manually via command line:

docker rm -f (docker ps -a -q)

If you use AddFromDockerfile() then it is recommended to prune images time to time:

docker image prune -f

Ideally, use the --filter option on the docker image prune commandline task. Simply add LABEL "CI_BUILD=True" in your Dockerfile, and force delete all images with that LABEL:

docker image prune -f --filter "CI_BUILD=True"

Release Notes

  • Add WSL2 Support
  • Migrated to net5.0 framework. Please reference 1.x.x packages if you need netstandard2.0 support. This version will continue getting critical fixes in branch.

testenvironment-docker's People

Contributors

a-z-hub avatar aharshkalep avatar deffiss avatar dependabot[bot] avatar dgvives avatar guddiny avatar hellevar avatar hughesappsheet avatar josephwoodward avatar jzabroski avatar kirylkiryanchykau avatar kumarnmanoj avatar lukino avatar oguzhaneren avatar romansp avatar taturevich avatar uladzimir-lashkevich avatar vhatsura avatar zaoralj avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

testenvironment-docker's Issues

Release a core version?

Would it be possible to release a version without dependencies to ElasticSearch and System.Data?

Use in Jenkins

Would it be possible to use this package in a Jenkins pipeline? As far as I know, Jenkins uses docker. Would this be a docker in docker situation? Or would it be more simple?

Postgres on CI/CD won't run

I use the package to run the test with PostgresDB. I followed your example here and it works fine, locally.

But when I run it on the ci/cd, it won't work.

Basically, it says :

System.Net.Http.HttpRequestException : Connection failed
---- System.Net.Internals.SocketExceptionFactory+ExtendedSocketException : Cannot assign requested address /var/run/docker.sock

image

Fix name of Kafka part

Invalid name Kafka project:
image
TestEnvironmen_b_t.Docker.Containers.Kafka -> TestEnvironment.Docker.Containers.Kafka

Change:

  • the name of the directory
  • project name
  • search other issues in the names

Unable to use ElasticsearchContainer in the project with NEST version of 7.x

The following exception is occurred during an attempt to bootstrap elasticsearch container in a project with 7.x NEST package:

System.TypeLoadException : Could not load type 'Nest.IClusterHealthResponse' from assembly 'Nest, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d'.
   at TestEnvironment.Docker.Containers.Elasticsearch.ElasticsearchContainerWaiter.Wait(ElasticsearchContainer container, CancellationToken cancellationToken)
   at TestEnvironment.Docker.Containers.Elasticsearch.ElasticsearchContainerWaiter.Wait(Container container, CancellationToken cancellationToken)
   at TestEnvironment.Docker.Container.WaitForReadiness(CancellationToken token)
   at TestEnvironment.Docker.Container.Run(IDictionary`2 environmentVariables, CancellationToken token)
   at TestEnvironment.Docker.DockerEnvironment.Up(CancellationToken token)

The issue is in breaking changes in NEST package, namely:

var health = await elastic.ClusterHealthAsync(
ch => ch
.WaitForStatus(WaitForStatus.Yellow)
.Level(Level.Cluster)
.ErrorTrace(true), cancellationToken);

in 7.x client, should be:

var health = await elastic.Cluster.HealthAsync(
                selector: ch => ch.WaitForStatus(WaitForStatus.Yellow).Level(Level.Cluster).ErrorTrace(true),
                ct: cancellationToken);

await elastic.DeleteIndexTemplateAsync("*");
await elastic.DeleteIndexAsync("*");

in 7.x client, should be:

await elastic.Indices.DeleteTemplateAsync("*", ct: token);
await elastic.Indices.DeleteAsync("*", ct: token);

I know updating the NEST package can be unreal, however, the ability to override cleaner and waiter for a container can solve the issue. It will allow using custom waiter with methods from the NEST 7.x package.

How are you promoting this?

This seems to be the most mature library for running docker integration tests in .NET, but I find nobody is talking about this in the .NET Community.

Unable to start two containers with similar names

During work under #31 I've found that the following setup is unable to start:

  1. Container #1 with test name
  2. Container #2 with test-1 name

The issue is in handling response from ListContainersAsync. It works like a regex and checks container names with startsWith in our case.

Default `Domainname` params for container cause error in docker 19 with userns-remap enable

Hi There,

This is Leo from BitBucket Pipeline 👋

Recently, we upgrade our docker version to 19 and receive bugs reported from our users who have their integration tests failed because of following error:

Error Message:
   Docker.DotNet.DockerApiException : Docker API responded with status code=BadRequest, response={"message":"OCI runtime create failed: container_linux.go:349: starting container process caused \"process_linux.go:449: container init caused \\\"write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: permission denied\\\"\": unknown"}

We did a bit of research for the error, and found that it is because in docker 19 introduce a new change which would lead to the error mentioned above when:

  1. docker-damon has config with 'userns-remap' = 'default'
  2. starting a container with docker run -ti --domainname my.domain some-image

You can find more details in here.

Whereas in testenvironment-docker, we found that it set the container domain name with container name by default:

var createParams = new CreateContainerParameters
{
Name = Name,
Image = $"{ImageName}:{Tag}",
AttachStdout = true,
Env = environmentVariables,
Hostname = Name,
Domainname = Name,
HostConfig = new HostConfig
{
PublishAllPorts = Ports == null
}
};

Wondering if testenvironment-docker can provide a way to override the Domainname option, then we can enable docker 19 for those users with minimum changes for their integration tests.

AddMssqlContainer extension error

Package versions
TestEnvironment.Docker 1.3.6
TestEnvironment.Docker.Containers.Mssql 1.0.6

If you execute the following code

using System.Threading.Tasks;
using TestEnvironment.Docker;
using TestEnvironment.Docker.Containers.Mssql;

namespace EnvironmentDockerIssue
{
    internal class Program
    {
        static async Task Main()
        {
            var environmentBuilder = new DockerEnvironmentBuilder();
            var env = environmentBuilder
                .UseDefaultNetwork()
                .SetName("testing")
                .AddMssqlContainer("mssql", "sapassword")
                .Build();

            await env.Up();
        }
    }
}

You will get the following:

Method not found: 'Void TestEnvironment.Docker.Container..ctor(Docker.DotNet.DockerClient, System.String, System.String, System.String, System.Collections.Generic.IDictionary`2<System.String,System.String>, Sy
stem.Collections.Generic.IDictionary`2<UInt16,UInt16>, Boolean, Boolean, TestEnvironment.Docker.IContainerWaiter, TestEnvironment.Docker.IContainerCleaner, Microsoft.Extensions.Logging.ILogger)'.
   at TestEnvironment.Docker.Containers.Mssql.MssqlContainer..ctor(DockerClient dockerClient, String name, String saPassword, String imageName, String tag, IDictionary`2 environmentVariables, IDictionary`2 ports, Boolean isDockerInDocker, Boolean reuseContainer
, ILogger logger)
   at TestEnvironment.Docker.Containers.Mssql.DockerEnvironmentBuilderExtensions.AddMssqlContainer(IDockerEnvironmentBuilder builder, String name, String saPassword, String imageName, String tag, IDictionary`2 environmentVariables, IDictionary`2 ports, Boolean
reuseContainer)
   at EnvironmentDockerIssue.Program.Main() in F:\EnvironmentDockerIssue\EnvironmentDockerIssue\Program.cs:line 12
   at EnvironmentDockerIssue.Program.<Main>()

The issue is not observed on the TestEnvironment.Docker 1.3.5
The sample to reproduce:
EnvironmentDockerIssue.zip

IAssemblyFixture issue. Tests execution does not end.

I’m having an issue when using IAssemblyFixture(from https://www.nuget.org/packages/Xunit.Extensions.AssemblyFixture) for EnvironmentFixture.
I works fine on my local machine, but not on CI pipeline. It does not throw any error it just runs and does not end.

Also IClassFixture works fine, but I want to share environment between several tests.

Here is GitHub action

runs-on: ubuntu-latest
timeout-minutes: 5

steps:
- uses: actions/[email protected]

- name: Setup .NET Core 5.0.x
  uses: actions/[email protected]
  with:
    dotnet-version: 5.0.x

- name: Restore dependencies
  run: dotnet restore

- name: Build
  run: dotnet build --configuration Debug --no-restore

- name: Test
  run: dotnet test --configuration Debug --no-build --verbosity normal

Usage

public class MyTests : IAssemblyFixture<EnvironmentFixture>
{
    private readonly EnvironmentFixture _environmentFixture;

    public MyTests(EnvironmentFixture environmentFixture)
    {
        _environmentFixture = environmentFixture;
    }.

EnveronmentFixture

public class EnvironmentFixture : IAsyncLifetime, IDisposable
{
    private readonly string _environmentName;
    private DockerEnvironment _environment;
    private IHost _host;

    public HttpClient TestClient { get; private set; }

    public EnvironmentFixture()
    {
        _environmentName = "test";
        Task.WaitAll(InitializeAsync());
    }

    public async Task InitializeAsync()
    {
        // Docker environment setup.
        _environment = CreateTestEnvironmentBuilder().Build();
        await _environment.Up();

        // API Test host setup
        _host = await CreateHostBuilder().StartAsync();

        TestClient = _host.GetTestClient();

        await OnInitialized(_environment.GetContainer<MongoContainer>("mongo"));
    }

I have mongo, redis and kafka containers registering following way

builder.AddContainer(name, "redis", "6-alpine");
builder.AddContainer(name, "docker.io/bitnami/kafka", "2-debian-10”);
builder.AddMongoContainer("mongo”);

Update to .NET6

I'm trying to run this with .NET 6. Lib works, but when I try to instantiate ServiceProvider I get a conflict error because TestEnvironment.Docker is using the 5.0.0 version of Microsoft Extension libraries.

Error CS0433 The type 'ServiceCollection' exists in both 'Microsoft.Extensions.DependencyInjection.Abstractions, Version=6.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60' and 'Microsoft.Extensions.DependencyInjection, Version=5.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'	Message.BrokerTest

Is there any plan on updating this?

Is it possible to have strong name assembly

Hey, @Deffiss.

Thanks a lot for such a great library. I was in DotNext in 2018 in Moscow and saw your presentation. It was awesome.

So, I'm using TestEnvironment.Docker package in a few projects and faced with an issue in one of the projects. Such a project has strong naming and has two targets: net461 and netcoreapp2.2. Unfortunately, TestEnvironment.Docker package cannot be used in net461 due to it isn't strong-named assembly:

System.IO.FileLoadException
Could not load file or assembly 'TestEnvironment.Docker, Version=1.2.2.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. A strongly-named assembly is required. (Exception from HRESULT: 0x80131044)

Also, the fresh guide from Microsoft about open-source libraries suggests signing libraries.
It would be very nice if you can do it.

SQL Server container fails to start on latest version of Docker (3.3.3) on MacOS

SQL Server container fails to start on latest version of Docker

Running this test fails with the following error when using the latest version of Docker Desktop (3.3.3) on a Mac.

I've tested this on a Windows machine and it looks like it doesn't return the IPv6 address.

Here's the stacktrace:

TestEnvironment.Docker.Tests.DockerEnvironmentTests.AddMsSqlContainer_WhenContainerIsUp_ShouldPrintMsSqlVersion

System.ArgumentException: An item with the same key has already been added. Key: 1433

System.ArgumentException
An item with the same key has already been added. Key: 1433
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](List`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at TestEnvironment.Docker.Container.RunContainerSafely(String[] environmentVariables, CancellationToken token) in /Users/joseph.woodward/Documents/git/Personal/testenvironment-docker/src/TestEnvironment.Docker/Container.cs:line 210
   at TestEnvironment.Docker.Container.Run(IDictionary`2 environmentVariables, CancellationToken token) in /Users/joseph.woodward/Documents/git/Personal/testenvironment-docker/src/TestEnvironment.Docker/Container.cs:line 84
   at TestEnvironment.Docker.DockerEnvironment.Up(CancellationToken token) in /Users/joseph.woodward/Documents/git/Personal/testenvironment-docker/src/TestEnvironment.Docker/DockerEnvironment.cs:line 43
   at TestEnvironment.Docker.Tests.DockerEnvironmentTests.AddMsSqlContainer_WhenContainerIsUp_ShouldPrintMsSqlVersion() in /Users/joseph.woodward/Documents/git/Personal/testenvironment-docker/test/TestEnvironment.Docker.Tests/DockerEnvironmentTests.cs:line 103
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_1.<<InvokeTestMethodAsync>b__1>d.MoveNext() in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\Runners\TestInvoker.cs:line 264
--- End of stack trace from previous location where exception was thrown ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in C:\Dev\xunit\xunit\src\xunit.execution\Sdk\Frameworks\ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in C:\Dev\xunit\xunit\src\xunit.core\Sdk\ExceptionAggregator.cs:line 90

image

Starting "Mongo" Container throws - failed to create shim: OCI runtime create failed

Error Trace


Message: 
Docker.DotNet.DockerApiException : Docker API responded with status code=InternalServerError, response={"message":"failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: sethostname: invalid argument: unknown"}


  Stack Trace: 
DockerClient.HandleIfErrorResponseAsync(HttpStatusCode statusCode, HttpResponseMessage response, IEnumerable`1 handlers)
DockerClient.MakeRequestAsync(IEnumerable`1 errorHandlers, HttpMethod method, String path, IQueryString queryString, IRequestContent body, IDictionary`2 headers, TimeSpan timeout, CancellationToken token)
ContainerOperations.StartContainerAsync(String id, ContainerStartParameters parameters, CancellationToken cancellationToken)
ContainerApi.CreateContainer(ContainerParameters containerParameters, CancellationToken cancellationToken)
ContainerApi.RunContainerAsync(ContainerParameters containerParameters, CancellationToken cancellationToken)
Container.RunAsync(CancellationToken cancellationToken)
DockerEnvironment.UpAsync(CancellationToken cancellationToken)
BookStoreServiceIntegrationTest.InitializeAsync() line 62

Code


     public async Task InitializeAsync()
        {
            _environment = CreateTestEnvironmentBuilder().Build();

            await _environment.UpAsync();

            _container = _environment.GetContainer<MongoContainer>("BookStoreServiceIntegrationTestMongo");
        }

        private IDockerEnvironmentBuilder CreateTestEnvironmentBuilder()
        {
            var builder = new DockerEnvironmentBuilder()
                .SetName("BookStoreServiceIntegrationTest")
                .AddMongoContainer(p => p with
                {
                    Name = "BookStoreServiceIntegrationTestMongo"
                });

            return builder;
        }

Environment:

  1. Windows
WindowsProductName WindowsVersion OsHardwareAbstractionLayer
------------------ -------------- --------------------------
Windows 10 Pro     2009           10.0.19041.1566

  1. WSL
 wsl --list
Windows Subsystem for Linux Distributions:
docker-desktop (Default)
docker-desktop-data

  1. Dotnet Environment - .net6

Binds during container creation

Good day,
In the older versions, you had the possibility to provide a list of Binds (mounts) when a container was created.
So I could mount some files or folders to a container (eg, it is very useful with databases when you have a sample data set, or when you need to provide some configs and etc.)
It was something like this

protected override CreateContainerParameters GetCreateContainerParameters(string[] environmentVariables)
{
    var parameters = base.GetCreateContainerParameters(environmentVariables);
    if (_binds != null && _binds.Any())
        parameters.HostConfig.Binds = _binds;

    return parameters;
}

I updated to the latest version and now I don't see this option, or maybe it was moved somewhere else
Could you provide some directions on how to do this now?
Thank you in advance.

Clarify licensing

Hi and thanks for the great library!

I would like to use it in my next project, is it possible to clarify under which license this code is distributed? Would be great to see this added into repository and nuget package. Thanks in advance!

Could not load MySqlConnector assembly due to old version

We are currently using version 1.3.1 of MySqlConnector in our project (latest is actually 1.3.3). However, TestEnvironment.Docker.Containers.MariaDB is using version 0.61.0. It seems that even with an assembly binding redirect in the app.config, the assembly cannot be loaded. I'm getting this error on environment.Up():

System.IO.FileLoadException: 'Could not load file or assembly 'MySqlConnector, Version=1.3.1.0, Culture=neutral, PublicKeyToken=d33d3e53aa5f8c92' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)'

Perhaps we can upgrade the nuget reference for MySqlConnector in TestEnvironment to the latest version?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.