GithubHelp home page GithubHelp logo

azure / azure-cosmos-db-emulator-docker Goto Github PK

View Code? Open in Web Editor NEW
142.0 58.0 43.0 25 KB

This repo serves as hub for managing issues, gathering feedback, and having discussions regarding the Cosmos DB Emulator Docker.

Home Page: https://learn.microsoft.com/en-us/azure/cosmos-db/how-to-develop-emulator?tabs=docker-linux%2Ccsharp&pivots=api-nosql

License: MIT License

Batchfile 40.31% PowerShell 27.18% Dockerfile 32.51%
cosmosdb cosmosdb-emulator

azure-cosmos-db-emulator-docker's Introduction

Azure Cosmos DB Emulator Docker Container

This repository contains the scripts required to install and run the Azure Cosmos DB Emulator as a Docker container.

You can fetch the image from Docker Hub by running docker pull mcr.microsoft.com/cosmosdb/windows/azure-cosmos-emulator.

About the Azure Cosmos DB service

Azure Cosmos DB is Microsoft's globally distributed, multi-model database. With the click of a button, Azure Cosmos DB enables you to elastically and independently scale throughput and storage across any number of Azure's geographic regions. It offers throughput, latency, availability, and consistency guarantees with comprehensive service level agreements (SLAs), something no other database service can offer.

About the Azure Cosmos DB emulator

The Azure Cosmos DB Emulator provides a local environment that emulates the Azure Cosmos DB service for development purposes. Using the emulator, you can develop and test your application locally, without creating an Azure subscription or incurring any costs. When you're satisfied with how your application is working in the Azure Cosmos DB Emulator, you can switch to using an Azure Cosmos DB account in the cloud.

Running inside Docker

The Azure Cosmos DB Emulator can be run on Docker Windows containers. The emulator does not work on Linux containers.

Once you have Docker for Windows installed, you can pull the Emulator image from Docker Hub by running the following command from your favorite shell (cmd.exe, PowerShell, etc.).

docker pull microsoft/azure-cosmosdb-emulator 

To start the image, run the following commands.

md %LOCALAPPDATA%\CosmosDBEmulatorCert 2>nul
docker run -v %LOCALAPPDATA%\CosmosDBEmulatorCert:c:\CosmosDBEmulator\CosmosDBEmulatorCert -P -t -i mcr.microsoft.com/cosmosdb/windows/azure-cosmos-emulator

The response looks similar to the following:

Starting Emulator
Emulator Endpoint: https://172.20.229.193:8081/
Master Key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==
Exporting SSL Certificate
You can import the SSL certificate from an administrator command prompt on the host by running:
cd /d %LOCALAPPDATA%\DocumentDBEmulatorCert
powershell .\importcert.ps1
--------------------------------------------------------------------------------------------------
Starting interactive shell

Closing the interactive shell once the Emulator has been started will shutdown the Emulator’s container.

Use the endpoint and master key in from the response in your client and import the SSL certificate into your host. To import the SSL certificate, do the following from an admin command prompt:

cd %LOCALAPPDATA%\CosmosDBEmulatorCert
powershell .\importcert.ps1

Developing against the emulator

See Developing against the emulator for how to connect to the emulator using one of the supported APIs/SDKs.

azure-cosmos-db-emulator-docker's People

Contributors

aliuy avatar arramac avatar microsoft-github-policy-service[bot] avatar microsoftopensource avatar milismsft avatar mominag avatar msftgits avatar robconery avatar sajeetharan avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

azure-cosmos-db-emulator-docker's Issues

Updated version doesn't seems to start

Hi,

I have update my emulator to the last version, I have downloaded it from :
https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator

Then the emulator start, everything looks ok, it try to open the URL : https://localhost:8081/_explorer/index.html

and there I got an error message : Error Code: INET_E_RESOURCE_NOT_FOUND

I have also try to connect with the Azure Storage Explorer (the new one) and I got
the same error message.

Please, can you help ?

Cosmos DB Linux Emulator fails to start on some Intel chips

Related to: actions/runner-images#5036 (comment)

The Cosmos DB Linux Emulator fails to start on some Intel chips.

lscpu output:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit

Byte Order: Little Endian
Address sizes: [46]
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz
Stepping: 7
CPU MHz: 2593.907
BogoMIPS: 87.81
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 64KiB
L1i cache: 64 KiB
L2 cache: 2 MiB
L3 cache: 35.8 MiB
NUMA node0 CPU(s): 0,1
Vulnerability Itlb multihit: KVM: Mitigation: VMX unsupported
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT Host state unknown
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt avx512cd avx512bw avx512vl xsaveopt xsavec xsaves md_clear
/proc/cpuinfo content:
/proc/cpuinfo

MicrosoftTeams-image (2)

Unable to skip SSL authentication when connecting to Emulator running in Linux Docker Container

Hi there
I am developing a C# .NET Core app (on windows) and want to connect to the cosmos emulator running in a linux docker container on docker desktop for windows. I don't want to use SSL authentication so I try to make use of the most recent CosmosClient of Microsoft.Azure.Cosmos package. However, this doesn't seem to work.

Steps to reproduce:

On a windows machine, pull the most recent image of the cosmos emulator

docker pull mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

Start up a new emulator container

docker run -p 8081:8081 -p 10251:10251 -p 10252:10252 -p 10253:10253 -p 10254:10254 -m 3g --cpus=2.0 --name=test-linux-emulator -e AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10 -e AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true -it mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

Navigate to https://localhost:8081/_explorer/index.html and download the sample app for C# .NET Core.

Open the sample app in Visual Studio and update the Nuget package Microsoft.Azure.Cosmos to Version 3.20.1.

Open the class Program.cs and change the method GetStartedDemoAsync to skip SSL authentication:

public async Task GetStartedDemoAsync()
{
   CosmosClientOptions cosmosClientOptions = new CosmosClientOptions()
   {
       HttpClientFactory = () =>
       {
           HttpMessageHandler httpMessageHandler = new HttpClientHandler()
           {
               ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
           };
           return new HttpClient(httpMessageHandler);
       },
       ConnectionMode = ConnectionMode.Gateway
   };

   // Create a new instance of the Cosmos Client
   this.cosmosClient = new CosmosClient(EndpointUri, PrimaryKey, cosmosClientOptions);
   await this.CreateDatabaseAsync();
   await this.CreateContainerAsync();
   await this.ScaleContainerAsync();
   await this.AddItemsToContainerAsync();
   await this.QueryItemsAsync();
   await this.ReplaceFamilyItemAsync();
   await this.DeleteFamilyItemAsync();
   await this.DeleteDatabaseAndCleanupAsync();
}

Then hit F5 to run the program.

Expected result
The program runs successfully.

Actual result
The program hangs in this.CreateDatabaseAsync();, never returning and never creating a database.

Multi Stage Build into Nano server

In order to keep image size small, I was hoping to create a multi-build Dockerfile, where I install cosmosDB Emulator into the servercore base image (using the msi), and copy the installation to a nano server image, where I can then spin up the emulator (since nano server does not support msi):

`FROM mcr.microsoft.com/windows/servercore:ltsc2019 AS dependencies
RUN Invoke-WebRequest -Uri https://aka.ms/cosmosdb-emulator -OutFile .\cosmosdb.msi; Start-Process msiexec.exe -Wait -ArgumentList '/I cosmosdb.msi /quiet'; rm .\cosmosdb.msi

FROM mcr.microsoft.com/dotnet/core/sdk as base
ARG cosmosDBPath="c:/Program Files/Azure Cosmos DB Emulator"
COPY --from=dependencies ${cosmosDBPath} ${cosmosDBPath}
And then within the script that is executed, have something to this effect:Start-Process -FilePath "C:\Program Files\Azure Cosmos DB Emulator\CosmosDB.Emulator.exe" -ArgumentList "/noui"`

However, this is working, and I see "No connection could be made because the target machine actively refused it." whenever I try to connect to "https://127.0.0.1:8081/_explorer/index.html".

Are there any suggestions for how I can get CosmosDB Emulator to work within a nano-server container?

Pull docker image on Mac OS

Just wondering is there any way to get docker image of cosmos DB and bring up a container service for it on Mac OS? Asking because we were using remote cosmos DB for dev before, but we have to build our project in local now for test purpose. In our case MacOS is the only choice and no VM is allowed.

Not able to pull collections in Web View

I am able to navigate to the emulator web portal, however, it fails when trying to retrieve collections. I am receiving the following errors:

Failed to load resource: net::ERR_CONNECTION_REFUSED: localhost:8081/dbs

Uncaught TypeError: Cannot read property 'queryDocuments' of undefined
    at Program.refreshCollection (built/Program.js:24)
    at HTMLSpanElement.<anonymous> (externals/knockout.js:89)
    at HTMLSpanElement.dispatch (jquery.js:9)
    at HTMLSpanElement.r.handle (jquery.js:9)

I believe this is occurring because when running the emulator in the Docker Container the URL is not localhost. In my case it was running on 172.26.69.202:8081.

Readme run cmd is wrong

I had to use this for it to work:

docker run -v %LOCALAPPDATA%\CosmosDBEmulatorCert:C:\CosmosDB.Emulator\bind-mount -P -t -i -m 2GB microsoft/azure-cosmosdb-emulator

Otherwise the cert import fails

How do I change the PartitionCount

When running in docker, it applies the standard 25-container limit.
How do I change my docker run command to increase this container limit?

Running Cosmos Emulator in background

I am attempting to start the Cosmos DB emulator during a build process, but I'm not having success when I try to start the emulator in detached code. The only way I can keep it running is to have a terminal connected (i.e., docker run -it).

What is the recommended way to start the emulator in the background?

Cosmos SDK experiences timeouts if mapped ports in docker-compose file differ

Receiving 408 timeouts with the cosmos C# SDK with specific port mapping configurations in a docker-compose file

docker compose that works

cosmos:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
    container_name: cosmos
    ports:
      - 8081:8081
      - 10251:10251
      - 10252:10252
      - 10253:10253
      - 10254:10254
    expose:
      - "8081"
      - "10251-10255"
    environment:
      - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=6
      - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false
      - AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=${LOCALIPADDRESS}

docker compose that does not work

cosmos:
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator
    container_name: cosmos
    expose:
      - "8081"
      - "10251-10255"
    ports:
      - 8082:8081
      - 10252:10251
      - 10253:10252
      - 10254:10253
      - 10255:10254
    environment:
      - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=6
      - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=false
      - AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=${LOCALIPADDRESS}

Function call that throws exception

var httpMessageHandler = new HttpClientHandler()
{
    ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};

CosmosClientOptions cosmosClientOptions = new CosmosClientOptions()
{
    HttpClientFactory = () => new HttpClient(httpMessageHandler),
    ConnectionMode = ConnectionMode.Gateway,
    Serializer = new CosmosJsonNetCustomSerializer(),
    RequestTimeout = TimeSpan.FromMinutes(3)
};

using CosmosClient client = new CosmosClient(cosmosEndpoint, COMSOS_KEY, cosmosClientOptions);

var db = await client.CreateDatabaseIfNotExistsAsync("DbName", 4000, null, CancellationToken.None);

The first docker-compose yaml works and the second does not. Receiving the following error

Unhandled exception. Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: RequestTimeout (408); Substatus: 0; ActivityId: 2f24681b-db18-44aa-afc8-e12ba02d8930; Reason: (GatewayStoreClient Request Timeout. Start Time UTC:03/29/2022 22:28:56; Total Duration:36015.4719 Ms; Request Timeout 20000 Ms; Http Client Timeout:180000 Ms; Activity id: 2f24681b-db18-44aa-afc8-e12ba02d8930;);

The reason I am looking to map the ports differently is i need to have two cosmos emulator docker containers running simultaneously.

Any assistance/insight would be greatly appreciated.

"Error: Password contains an illegal unescaped character"

My code is (I've changed localhost port to 8000 & mongodb port to 10000 with CosmosDB.Emulator.exe):

var url = "mongodb://localhost:C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==@localhost:10000/admin?ssl=true"

MongoClient.connect(url, function(err, db) { 
    if (err) throw err; // Return error but run the program: $ return console.error(err);
   ...
};

And it produces this error:

Error: Password contains an illegal unescaped character
    at parseConnectionString (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\url_parser.js:287:13)
    at parseHandler (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\url_parser.js:119:14)
    at module.exports (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\url_parser.js:25:12)
    at connect (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\mongo_client.js:869:3)
    at connectOp (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\mongo_client.js:269:3)
    at executeOperation (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\utils.js:409:22)
    at MongoClient.connect (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\mongo_client.js:260:10)
    at Function.MongoClient.connect (F:\Chatbots\Code\test\botframework-demo\node_modules\mongodb\lib\mongo_client.js:481:22)
    at Object.<anonymous> (F:\Chatbots\Code\test\botframework-demo\test.js:145:13)
    at Module._compile (module.js:635:30)

I've also tried to change url for encodeURIComponent(url), but it produces the next error:

Error: Invalid schema, expected mongodb or mongodb+srv

What I need to do?

Linux emulator fails on macOS M1 even with x64 virtualization

Understandably, you do not support M1 chips and it is questionable if supporting M1 chips is worth prioritizing; however, somehow the container fails on M1 even when using x64 virtualization, as demonstrated in #17.

At a minimum, can we have the container work with x64 emulation?

Re-enabling Data Explorer with custom master key

To the best of my knowledge and understanding, changing the master key renders the data explorer ui useless.

Could modifying the startemu.cmd such that the key used is mapped to an environment variable allow the Data Explorer to access the Cosmos db again? Is the key baked into the Data Explorer code or does it derive the key from sources similar to those used in the startemu.cmd script?

Be able to export rsa-2048 cert or higher

It looks like the certificate that this generates is only rsa-1024, which lately isn't good enough for a lot of communication. For example Azure Core Tools running in a Linux environment isn't able to create a secure network connection with the emulator because the certificate is too weak. Is it possible to make the cert more secure or to at least make it configurable?

azure-cosmos-db-emulator-docker unable to connect Robo 3t or mongo api. Works fine with azure document db api

I can able to install the docker container in my local , But when trying to connect with Mongo 3T or with my .net application which uses mongo client nuget package. It throws timeout error .

"A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "Unknown", State : "Disconnected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "172.29.117.101:10255" }", EndPoint: "172.29.117.101:10255", State: "Disconnected", Type: "Unknown", LastUpdateTimestamp: "2019-08-28T11:48:28.0587550Z" }] }."

Only I have problem with docker container . when i do the same with Cosmos Db emulator everything works fine in local.
I have tried connecting with azure document db it works, But facing problem when i trying the same with mongo api which runs inside the docker container .

Basically Am trying to run integration test in my azure pipeline by adding azure cosmos db emulator task, which unfortunately not work with mongo api alone .

Be able to use custom certificate

It would be easier to use the CosmosDB emulator docker image for local development if we could specify a custom certificate to use, instead of having to install the generated one into the local certificate store (and re-install it when it changes).

In our case, we would like to be able to use https://github.com/FiloSottile/mkcert to generate a locally trusted ca, then issue a cert for CosmosDB to use.

I notice that you can use a custom certificate using AZURE_COSMOS_EMULATOR_CERTIFICATE. Unfortunately from testing it looks like the emulator will only work if its a self signed certificate issued by localhost.

For some context, we run the emulator in docker-compose (using the linux image) for local development. Our developers use a mix of Linux, Mac and Windows. Automating the import of the certificate across all 3 platforms took a lot longer than it would have with mkcert, and requires root/admin.

Have problem when try to start emulator i docker compose

Use that configuration in docker compose

cosmos:
    image: "microsoft/azure-cosmosdb-emulator"
    mem_limit: 2G
    ports:
      - "8081:8081"
      - "8900:8900"
      - "8901:8901"
      - "8979:8979"
      - "10250:10250"
      - "10251:10251"
      - "10252:10252"
      - "10253:10253"
      - "10254:10254"
      - "10255:10255"
      - "10256:10256"
      - "10350:10350"
    networks:
      net:
        ipv4_address: 10.5.0.3
    volumes:
      - C:\Users\Illia_Lubenets\AppData\Local\CosmosDBEmulator\bind-mount:C:\CosmosDB.Emulator\bind-mount

and receive that error

cosmos_1   | Exception setting "BufferWidth": "The handle is invalid.
cosmos_1   | "
cosmos_1   | At C:\CosmosDB.Emulator\Start.ps1:37 char:1
cosmos_1   | + [Console]::BufferWidth = 32766
cosmos_1   | + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cosmos_1   |     + CategoryInfo          : NotSpecified: (:) [], SetValueInvocationExceptio 
cosmos_1   |    n
cosmos_1   |     + FullyQualifiedErrorId : ExceptionWhenSetting
cosmos_1   |  
cosmos_1   | Transcript started, output file is C:\CosmosDB.Emulator\bind-mount\Diagnostics\Transcript.log

Azure Cosmos DB Emulator cannot be installed on AD domain controller machine

The Azure Cosmos DB Enumerator is working well when running on regular computers, but when I tried to install in on one domain controller machine, I got following error:

Application: DocumentDB.StartupEntryPoint.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Security.Principal.IdentityNotMappedException
at System.Security.Principal.NTAccount.Translate(System.Security.Principal.IdentityReferenceCollection, System.Type, Boolean)
at System.Security.Principal.NTAccount.Translate(System.Type)
at Microsoft.Azure.Documents.Common.SecurityUtil.AccountNameToSid(System.String)
at Microsoft.Azure.Documents.Services.ServiceStartupEntryPoint.ServiceStartupEntryPoint.GetSecurityGroupSid(System.String)
at Microsoft.Azure.Documents.Services.ServiceStartupEntryPoint.ServiceStartupEntryPoint.GatewayInit(System.Fabric.CodePackageActivationContext)
at Microsoft.Azure.Documents.Services.ServiceStartupEntryPoint.ServiceStartupEntryPoint.Main(System.String[])

I checked the code, the problem happens to the following linke
string securityGroupSid = GetSecurityGroupSid("DocDBGatewayService");

which internally try to find the SID for account \DocDBGatewayservice, where on a domain controller, the machine name is not valid account scope. Since the DocDBGatewayservice is not present anyway, so we should try catch and ignore such exception.

Ability to Support GremlinEndpoint

Are there any plans to support the GremlinEndpoint within the docker Docker Emulator?

I've edited the Start.ps1 to start the emulator with -EnableGremlinEndpoint, but after that to Container stops starting.

Unable to create a collection

Hi,

I setup the Cosmos DB Emulator on Docker for Windows with the intend to use it for Graph database with Gremlin API.

According to the docs I shall create a database and a collection, see here.
However, I cannot create a collection, only databases and containers, which results only in SQL databases as far as I understand:

image

How can I solve this? Makes the emulator pretty useless for my case.
Issue may be related to #7

Thanks!

Azure cosmos baseuri port change for docker container running on macOS

Trying to run cosmos emulator container on port external 8082 port, but when using java sdk for same, it connects to port 8081 internally and gives error like :
14:19:32.373 [reactor-http-kqueue-3] ERROR com.azure.cosmos.implementation.RxGatewayStoreModel - Network failure
io.netty.channel.ConnectTimeoutException: connection timed out: /10.88.0.4:8081
at io.netty.channel.kqueue.AbstractKQueueChannel$AbstractKQueueUnsafe$2.run(AbstractKQueueChannel.java:563) ~[netty-transport-classes-kqueue-4.1.77.Final.jar:4.1.77.Final]

Is it not possible to change the port like we can do for windows cosmos emulator application.Please suggest

CosmosDB Explorer Issues

CosmosDB explorer is no longer showing collections, users, stored procedures etc. the browser shows a single 404 error for https://...:8081/_explorer/externals/bootstrap.min.css.map

It's not possible to use development setup with Visual Studio using Docker Compose and Cosmos Db

This is my current local development scenario:

  • Dotnet core 6.0 application
  • Visual Studio 2022 using a multi-container app with Docker Compose
  • On docker-compose.yml I do the setup for all necessary resources to run my project, like:
    • Redis
    • Rabbit Mq
    • Sql Server
    • Cosmos Db
  • I'm using a windows machine running linux containers
  • I have some integration tests that I run using Visual Studio Test Explorer

I'm using this Docker image for Cosmos Db Emulator on Linux mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator, this is my issue: If I don't set the environment variable AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE then it's possible to access the cosmos db from the application, but it's not possible to access from my Integration Tests, otherwise if I set the variable AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE with the value 127.0.0.1 then it's possible to access from my integration tests but it is NOT possible to access from my application, this is the docker-compose.yml file:

version: '3.7'

services:

  cosmosdb:
    container_name: cosmos-db
    image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest
    cpus: 2.0
    mem_limit: 3g
  
  sqldata:
    container_name: sqlserver-db
    image: mcr.microsoft.com/mssql/server:2019-latest

  redisdata:
    container_name: redis-data
    image: redis:alpine

  rabbitmq:
    container_name: rabbit-mq
    image: rabbitmq:3-management-alpine

  service-api:
    container_name: service-api
    image: ${REGISTRY:-abc}/service.api:${PLATFORM:-linux}-${TAG:-latest}
    build:
      context: .
      dockerfile: src/Service.Backend.Web.API/Dockerfile
      network: host
    depends_on:
      - sqldata
      - redisdata
      - rabbitmq
      - cosmosdb

And here it's my docker-compose.override.yml file:

version: '3.7'

services:

  cosmosdb:
    environment:
       - AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
       - AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
       - AZURE_COSMOS_EMULATOR_GREMLIN_ENDPOINT=true
       # If unncoment this line then it's possible to access from outside,
       # but the application can't reach the database
       #- AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=127.0.0.1
    ports:
      - "8081:8081"
      - "8901:8901"
      - "8902:8902"
      - "10251:10251"
      - "10252:10252"
      - "10253:10253"
      - "10254:10254"
      - "10255:10255"
      - "10350:10350"
    volumes:
       - service-cosmosdbdata:/data/db

  sqldata:
    environment:
      - SA_PASSWORD=Pass@word
      - ACCEPT_EULA=Y
    ports:
      - "5434:1433"
    volumes:
      - service-sqldata:/var/opt/mssql

  redisdata:
    ports:
      - "6380:6379"
    volumes:
      - service-redisdata:/var/opt/redis
  
  rabbitmq:
    ports:
      - "15672:15672"
      - "5672:5672"
    volumes:
      - service-rabbit:/var/opt/rabbitmq

  service-api:
    environment:
      - ASPNETCORE_ENVIRONMENT=Development
      - ASPNETCORE_URLS=https://+:443;http://+:80
      - ASPNETCORE_Kestrel__Certificates__Default__Password=Pass@word
      - ASPNETCORE_Kestrel__Certificates__Default__Path=/https/aspnetapp.pfx
      - ConnectionStrings__DefaultConnection=Server=sqldata;Database=service-db;User Id=sa;Password=Pass@word
      - ConnectionStrings__EventStoreConnection=Server=sqldata;Database=eventstore-db;User Id=sa;Password=Pass@word
      - ConnectionStrings__HistoryDbConnection=Server=sqldata;Database=history-db;User Id=sa;Password=Pass@word
      - ConnectionStrings__CosmosDb__ServiceEndpoint=https://cosmosdb:8081
      - ConnectionStrings__CosmosDb__AuthKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==
      - ConnectionStrings__Redis=redisdata
      - EventBus__ServiceBusEnabled=false
      - EventBus__RabbitMQ__Connection=rabbitmq
    ports:
      - "5001:80"
      - "5002:443"
    volumes:
      - ~/.aspnet/https:/https:ro

volumes:
  service-rabbit:
    external: false
  service-sqldata:
    external: false
  service-cosmosdbdata:
    external: false
  service-redisdata:
    external: false

And here it's a sample of a dumb test that works only whe set the variable AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE:

[Theory]
[InlineData("https://localhost:8081")]
[InlineData("https://host.docker.internal:8081")]
[InlineData("https://127.0.0.1:8081")]
public async Task TestWithLocalhost(string accountEndpoint)
{
    var cosmosClient = new CosmosClient(
        accountEndpoint,
        "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==",
        new CosmosClientOptions
        {
            SerializerOptions = new CosmosSerializationOptions
            {
                IgnoreNullValues = true,
                Indented = true,
                PropertyNamingPolicy = CosmosPropertyNamingPolicy.CamelCase
            },
            HttpClientFactory = () =>
            {
                HttpMessageHandler httpMessageHandler = new HttpClientHandler()
                {
                    ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator,
                };

                return new HttpClient(httpMessageHandler);
            },
            ConnectionMode = ConnectionMode.Gateway,
        });

        var cancellationToken = new CancellationTokenSource();

        cancellationToken.CancelAfter(TimeSpan.FromSeconds(30));

        var result = await cosmosClient.CreateDatabaseIfNotExistsAsync("UnitTest", cancellationToken: cancellationToken.Token);

        cancellationToken.Dispose();

        result.StatusCode.Should().BeOneOf(HttpStatusCode.Created, HttpStatusCode.OK);
}

Any sugetions on how can I manage this situation? I also tried to Cosmos Db emulator installed on my local machine, but I had the same problem, the integration tests can access but the application can not.

Docker Container - Excessive CPU usage

According to the emulator release logs, version 2.14.6 fixed an issue with high CPU usage. I'm using the dockerized version of the emulator and believe I am suffering under this problem as well, as the docker-image has been created in January. In the current state, the emulator is unuseable, because it leads to > 95% cpu usage and system freezes. I'm not even doing anything with the emulator, just starting the emulator with 4 partitions is enough to produce this behavior.

https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator-release-notes

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 2
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cc61520f4cd876b86e77edfeb88fbcd536d1f9d
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.16.3-microsoft-standard-WSL2
 Operating System: Ubuntu 20.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 24.86GiB
 Name: PN1030UZ2603124
 ID: NM3I:CLYP:TPND:KIAR:2PKN:7LGQ:MHW5:X2MF:3624:O2ZJ:RKG6:DDQO
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
   Base: 10.0.1.0/24, Size: 24

Linux Image

any chance a Linux Image could be added to the backlog?

I have a Raspberry Pi Cluster and would love to run it on there.

unable to connect

I'm running the docker image as described, however I'm unable to connection to the CosmosDB server running within docker on any exposed port?

Error: The evaluation period has expired.

I have just pulled the image and tried to run it but getting the error below:

Image: mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

Error:
2022-11-15 15:06:02 Error: The evaluation period has expired.
2022-11-15 15:06:02 ./cosmosdb-emulator: ERROR: PAL initialization failed. Error: 104

The only reference to this I could find online is: https://stackoverflow.com/questions/68187668/cosmos-db-emulator-for-linux-docker-this-is-an-evaluation-version-there-are-1

I attempted to remove the containers/images and pull the latest image again to no avail

[Question] Where is Cosmos DB "Connected Service" VS functionality maintained?

I wanted to ask for a new ConnectedService option for Cosmos in Visual Studio that only used the Cosmos DB Emulator but not the docker-hosted version, but I have no idea where that piece of code/extension is currently maintained (so that I could go there and create the issue).

I know this project has some relation to that functionality since it contains the docker images used on the "CosmosDB Emulator on Docker" option that exists currently in Connected Services, so I figured someone over here could provide some guidance.

Cosmos emulator not starting in GitHub Runner ubuntu 20.04 and 22.04

Right now, the docker image cannot run in a GitHub action with ubuntu 20.04 and ubuntu 22.04. It doesn't initialize for some reason. The logs only says:

This is an evaluation version.  There are [113] days left in the evaluation period.
Shutting Down
Shut Down

I created a public repository to show the problem. There is only a docker-compose.yml with a cosmosdb service and a github workflow that up services, sleep 60s and check the logs of the container. So you can see the action log.
There we can see that only works in Ubuntu 18.04.

https://github.com/DrDonoso/CosmosDBDockerGHA/actions/runs/3966547297/jobs/6797477850

Thank you in advance.

MongoDB API on Linux not working

When running the documented docker command found on
https://docs.microsoft.com/en-us/azure/cosmos-db/linux-emulator?tabs=mongodb-api%2Cssl-netstd21#run-the-linux-emulator-on-linux-os

docker run
--publish 8081:8081
--publish 10251-10254:10251-10254
--memory 3g --cpus=2.0
--name=test-linux-emulator-mongo
--env AZURE_COSMOS_EMULATOR_PARTITION_COUNT=10
--env AZURE_COSMOS_EMULATOR_ENABLE_DATA_PERSISTENCE=true
--env AZURE_COSMOS_EMULATOR_IP_ADDRESS_OVERRIDE=$ipaddr
--env AZURE_COSMOS_EMULATOR_ENABLE_MONGODB_ENDPOINT=4.0
--interactive
--tty
mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator

The container just keeps restarting...
Looking at the container logs, I see the following message.

Invalid endpoint type enabled; only SQL endpoint type is supported.

I am trying to run the on WSL2 configured for Linux containers.

Running Emulator gives Starting Emulator and nothing else

When I run the command:

docker run -v %LOCALAPPDATA%\CosmosDBEmulatorCert:c:\CosmosDBEmulator\CosmosDBEmulatorCert -P -t -i microsoft/azure-cosmosdb-emulator

I get:

Starting Emulator

after waiting about 3 minutes or so for further output, the process exits.

I checked %LOCALAPPDATA%\CosmosDBEmulatorCert and the folder is empty.

Any ideas?

ChangeFeedProcessorBuilder.handleAllVersionsAndDeletesChanges doesn't received any messages

I created a junit with the code from https://github.com/Azure-Samples/azure-cosmos-java-sql-api-samples/blob/fb3cee8e3d8faa69d97f6a61d4bf0633992426d0/src/main/java/com/azure/cosmos/examples/changefeed/SampleChangeFeedProcessorForAllVersionsAndDeletesMode.java

but when I run it with Cosmos DB emulator, I don't received messages with ChangeFeedProcessor and handleAllVersionsAndDeletesChanges.

I'm using the latest docker image version : mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:latest

I created a project to reproduce the issue.

I have few tests

src/test/java/com/example/demoredis/CosmosDBTests.java (with .handleChanges and it works)
src/test/java/com/example/demoredis/CosmosDBFeed3Tests.java : SampleChangeFeedProcessorForAllVersionsAndDeletesMode (doesn't work)

here the logs for CosmosDBFeed3Tests

[debug.log](https://github.com/Azure/azure-cosmos-db-emulator-docker/files/10297050/debug.log)

here my source code
demo-cosmos-feed-update.zip

unable to use linux emulator together with dotnet testrunner (net6)

we have tried for several days to get a stable execution environment for system tests with the cosmosdb linux emulator
We have tried to run the emulator in ubuntu-18.04 devops pipelines as well as on wsl2 ubuntu linux.

It seems like the emulator starts up fine, we have been able to run the dotnet core test program as well as a small program
written to test both EntityFramwork Core access as well as direct cosmosdb client access.
But our unittests and systemtests that are executed via dotnet test fails consistently, most of the time we get 408 timeout
but we can also get 503 (especially after a few 408).
We run the test against a newly created cosmosdb container to make sure that we have as well defined environment as possible. we also remove and adds the new certificate (we see ssl failures if we don't update the cert)

Is there differences between running dotnet run vs dotnet test that can explain the issues we see?

Image doesn't run on windows server 2019 docker

Tried to run this image on windows server 2019 in Azure and received:

docker pull microsoft/azure-cosmosdb-emulator

C:\Program Files\Docker\docker.exe: Error response from daemon: CreateComputeSystem d78c0667603b0fa05a008d65f06b59fbd93eceb4f26a349498df3f5d66615390: The container operating system does not match the host operating system.
(extra info: {"SystemType":"Container","Name":"d78c0667603b0fa05a008d65f06b59fbd93eceb4f26a349498df3f5d66615390","Owner":"docker","VolumePath":"\\\\?\\Volume{fd39e1ff-86a8-4edc-bdf3-43a68fbcff28}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\d78c0667603b0fa05a008d65f06b59fbd93eceb4f26a349498df3f5d66615390","Layers":[{"ID":"e703197d-7eb6-516c-9c8e-ee8277d36299","Path":"C:\\ProgramData\\docker\\windowsfilter\\a54a085f0171eb62a5e6283e1a718a140744709741890380712abdde99879a04"},{"ID":"96c0f581-b443-57f3-9a91-2fb9dc335b98","Path":"C:\\ProgramData\\docker\\windowsfilter\\63556e8392e32864b635c34a4845c3f5488f3d178a515c258b1aac9e12b8c576"},{"ID":"e73eeb53-762e-51bf-a2a5-8413bd57c1f3","Path":"C:\\ProgramData\\docker\\windowsfilter\\2e05f190b6d00e60db94463636957b1e8b9948f9a86bcd185e2e085a9547a190"},{"ID":"522af03d-ca75-566a-a33a-9da0b3ad35da","Path":"C:\\ProgramData\\docker\\windowsfilter\\1a307c4d49ed20f9a2c45d7ef23f5f76675e54eacb5241b6e765f01c28bab275"},{"ID":"35488ffd-95b6-5d50-a61f-54d28c5918b4","Path":"C:\\ProgramData\\docker\\windowsfilter\\192b7d18bb06717da5791104e90b202fce56c821b9fbf5c81c9b61094b3cfe62"},{"ID":"6c23280e-815d-5675-abc9-a7573e57286e","Path":"C:\\ProgramData\\docker\\windowsfilter\\11122141f17ccbe5d96a1af7c4cb0518776f2bb8533c0d89b1d895e08f73a179"},{"ID":"d4ae93bf-fe12-5cbf-a1d0-8409575739b4","Path":"C:\\ProgramData\\docker\\windowsfilter\\7ae8a8f0a112b3978170a2c62d8ac856f711a93aa14b46be642890024d7ce083"},{"ID":"2e9c5c7d-0775-53c8-80c5-f66513ca4aa9","Path":"C:\\ProgramData\\docker\\windowsfilter\\3473a5af443d9925ab51c89ef6dfe39c0a6f509baff1a7997452af5d68de8166"},{"ID":"5de1c788-d71b-5e9a-b246-7a0c484277f5","Path":"C:\\ProgramData\\docker\\windowsfilter\\f406143b1a024901ae9382e0fe59d61965e097a9ef5974204149dfff9e850d6a"},{"ID":"bb57300d-edb4-5ba5-b1a9-16f07cb66100","Path":"C:\\ProgramData\\docker\\windowsfilter\\3bd5d64675da336dfb59e50151646cb8e691faf5bbc977d19192c9205bdd156c"},{"ID":"f2e4fde9-ca9b-5821-a9c4-a01edd9aa269","Path":"C:\\ProgramData\\docker\\windowsfilter\\01e41ea7fdb63c8c1b0558d1159fc9b8c7b957b2a780fb13c0ca048dfe3a926b"},{"ID":"3069bb7f-f69d-53af-9a3e-441aa2986036","Path":"C:\\ProgramData\\docker\\windowsfilter\\42e6331c13fae9de3728a473b47cb250d5f603f63f2e02c09e55d3e51e7493ef"}],"MemoryMaximumInMB":2048,"HostName":"d78c0667603b","MappedDirectories":[{"HostPath":"C:\\Users\\kepleradmin\\AppData\\Local\\CosmosDBEmulator\\bind-mount","ContainerPath":"C:\\CosmosDB.Emulator\\bind-mount","ReadOnly":false,"BandwidthMaximum":0,"IOPSMaximum":0,"CreateInUtilityVM":false}],"HvPartition":false,"EndpointList":["1C770668-47E9-40A2-9451-1DE96A034ADB"],"AllowUnqualifiedDNSQuery":true}).

I would guess the image needs to be rebuilt to support the latest windows server OS?

Support for Linux

Hi

Is there any plan to support Linux for the emulator, its a very useful tool for writing integration tests and CI/CD pipelines. For devs that aren't just tinkering around the simulator has almost no value.

ServiceUnavailable (503); Substatus: 20003

Hi guys,

First of all, I want to thank you for the emulator and your work on it 👍

We use this emulator in our temporary Kubernetes environments for integration tests. (A Kubernetes cluster/AKS is created, pods created, services are deployed, integration tests are run, and the environment is dropped. All this in the Pull Request process - CI)

Unfortunately, it sometimes happens that the integration tests for one of our services that connect to this emulator fail. The only thing we can detect is that we get an exception: Microsoft.Azure.Cosmos.CosmosException: Response status code does not indicate success: ServiceUnavailable(503); Substatus: 20003; ActivityId: a4851255-27a6-4900-979b-4a47746a8cbc; Reason:().

The CosmosDb pod does not report any failures.

Unfortunately, we don't know how to proceed, what needs to be found out, and how to solve it. It's unpredictable.

Can you please help us?
Thanks a lot 🙏

Certificate thumbprint changes every time the container is re-started

I'm using this command to create the container:

docker create --name azure-cosmosdb-emulator --memory 2GB --mount "type=bind,source=${env:LOCALAPPDATA}\azure-cosmosdb-emulator.hostd,destination=C:\CosmosDB.Emulator\bind-mount" --tty -p 8081:8081 -p 8900:8900 -p 8901:8901 -p 8979:8979 -p 10250:10250 -p 10251:10251 -p 10252:10252 -p 10253:10253 -p 10254:10254 -p 10255:10255 -p 10256:10256 -p 10350:10350 microsoft/azure-cosmosdb-emulator

I then run:

pushd $env:LOCALAPPDATA\azure-cosmosdb-emulator.hostd
powershell .\importcert.ps1

Then stop and re-start the container (e.g. what happens if I shut my laptop down at the end of the day then start it up again the following day):

docker stop azure-cosmosdb-emulator
docker start azure-cosmosdb-emulator

When it restarts, it recreates the certificate (different thumprint), which I then have to re-import again. As an added complication, it's tricky to script/automate this as it doesn't generate this new certificate until the first request to the emulator.

This is an example of the Transcript.log after the restart:

**********************
Windows PowerShell transcript start
Start time: 20190423112804
Username: User Manager\ContainerAdministrator
RunAs User: User Manager\ContainerAdministrator
Machine: 713CE3FA5311 (Microsoft Windows NT 10.0.14393.0)
Host Application: powershell.exe -NoExit -NoLogo -Command C:\CosmosDB.Emulator\Start.ps1
Process ID: 1844
PSVersion: 5.1.14393.2636
PSEdition: Desktop
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.14393.2636
BuildVersion: 10.0.14393.2636
CLRVersion: 4.0.30319.42000
WSManStackVersion: 3.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: 1.1.0.1
**********************
Transcript started, output file is C:\CosmosDB.Emulator\bind-mount\Diagnostics\Transcript.log

Until the first request is made, it pauses at this point.


    Directory: C:\CosmosDB.Emulator\bind-mount


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        4/23/2019  11:29 AM            513 CosmosDbEmulatorCert.cer

Key   : GremlinEndpoint
Value : {http://713ce3fa5311:8901/, http://192.168.127.20:8901/}
Name  : GremlinEndpoint


Key   : TableEndpoint
Value : {https://713ce3fa5311:8902/, https://192.168.127.20:8902/}
Name  : TableEndpoint


Key   : Key
Value : C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==
Name  : Key


Key   : Version
Value : 2.2.1.0
Name  : Version


Key   : IPAddress
Value : 192.168.127.20
Name  : IPAddress


Key   : Emulator
Value : CosmosDB.Emulator
Name  : Emulator


Key   : CassandraEndpoint
Value : {tcp://713ce3fa5311:10350/, tcp://192.168.127.20:10350/}
Name  : CassandraEndpoint


Key   : MongoDBEndpoint
Value : {mongodb://713ce3fa5311:10255/, mongodb://192.168.127.20:10255/}
Name  : MongoDBEndpoint


Key   : Endpoint
Value : {https://713ce3fa5311:8081/, https://192.168.127.20:8081/}
Name  : Endpoint

Intermittent SSL failures in Azure pipeline

Hi, I'm running integrations tests within my Azure pipeline through the use of docker compose.

As part of this I'm spinning up the Linux cosmos Db emulator, and then running update-ca-certificates --fresh in order to allow SSL connections to cosmos from other containers within my test environment (as per the instructions here: https://learn.microsoft.com/en-us/azure/cosmos-db/docker-emulator-linux?tabs=sql-api%2Cssl-netstd21#run-the-linux-emulator-on-linux-os).

This works every time when running these tests locally, but when running on an Azure hosted agent (I've tried ubuntu 18.04, 20.04 and 22.04) I'm finding the trusting of the certs failing randomly - sometimes works fine, other times I receive the warning rehash: warning: skipping emulatorcert.pem,it does not contain exactly one certificate or CRL, which ultimately results in SSL exceptions and the tests failing to run.

Is there anyway to prevent these random failures and make the pipeline more reliable?

Persistent database using docker container

Emulator is unable to start when binding data folder nor using volumes to handle persistent database

- db_data:C:\Users\ContainerAdministrator\AppData\Local\CosmosDBEmulator
- type: bind
source: .\cosmosdb_data
target: C:\Users\ContainerAdministrator\AppData\Local\CosmosDBEmulator

From transcript you can find that it blocks on Start-CosmosDbEmulator cmdlet

?**********************
Windows PowerShell transcript start
Start time: 20200918113722
Username: User Manager\ContainerAdministrator
RunAs User: User Manager\ContainerAdministrator
Machine: COSMOSDB (Microsoft Windows NT 10.0.14393.0)
Host Application: powershell.exe -NoExit -NoLogo -Command Remove-Item -Path C:\CosmosDB.Emulator\bind-mount\*.pem; C:\CosmosDB.Emulator\Start.ps1; certutil -f -encode C:\CosmosDB.Emulator\bind-mount\CosmosDbEmulatorCert.cer C:\CosmosDB.Emulator\bind-mount\CosmosDbEmulatorCert.pem
Process ID: 1844
PSVersion: 5.1.14393.3471
PSEdition: Desktop
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.14393.3471
BuildVersion: 10.0.14393.3471
CLRVersion: 4.0.30319.42000
WSManStackVersion: 3.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: 1.1.0.1
**********************
Transcript started, output file is C:\CosmosDB.Emulator\bind-mount\Diagnostics\Transcript.log
INFO: Set partition count to 100
INFO: Stop-CosmosDbEmulator
INFO: Start-CosmosDbEmulator -AllowNetworkAccess -NoFirewall -NoUI -Key C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==  -Consistency Session -Timeout 300 -PartitionCount 100
Wait-CosmosDbEmulator : The emulator failed to reach Running status within 300 seconds
At C:\Program Files\Azure Cosmos DB
Emulator\PSModules\Microsoft.Azure.CosmosDB.Emulator\Microsoft.Azure.CosmosDB.Emulator.psm1:550 char:12
+     [void](Wait-CosmosDbEmulator -Status Running -Timeout $Timeout)
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Wait-CosmosDbEmulator
Wait-CosmosDbEmulator : The emulator failed to reach Running status within 300 seconds
At C:\Program Files\Azure Cosmos DB Emulator\PSModules\Microsoft.Azure.CosmosDB.Emulator\Microsoft.Azure.CosmosDB.Emulator.psm1:550 char:12
+     [void](Wait-CosmosDbEmulator -Status Running -Timeout $Timeout)
+            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Wait-CosmosDbEmulator

Maybe you are aware about another way to do this?

Question: Is there a mechanism to see the database is started

Is there a command or mechanism that we could run as a health check or other means to make sure the database inside the container is actually started. So that one could set the depends_on in a docker compose file for example, or in other environments like when using Testcontainers library for dot-net to wait until the database is actually started.

Running with Azure Storage Emulator

Is it possible to use the Azure Storage Emulator to hold the Web Jobs with this? It looks like the config for AzureWebJobsStorage either allow local dev (assuming installed on the running device) or in Azure. The connection string I have tried is below and using the Storage Emulator in Docker from https://github.com/tg123/Docker-AzureStorageEmulator. Be great to be able to merge this.

"AzureWebJobsStorage": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;TableEndpoint=http://localhost:10001/devstoreaccount1;QueueEndpoint=http://localhost:10002/devstoreaccount1;",
"AzureWebJobsDashboard": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;TableEndpoint=http://localhost:10001/devstoreaccount1;QueueEndpoint=http://localhost:10002/devstoreaccount1;",

Use Nano Server image as base instead of Windows Server Core

The disk space/bandwidth requirements for Windows Server Core are pretty extreme compared to Nano Server (About 9Gb vs 600Mb). It would be great to see this docker image use Nano Server as it's base instead.

I realise that Nano Server does not support MSI's (which is how the emulator is shipped) which is probably why the image uses Windows Server Core, but it must be possible to obtain an xcopy/binaries only version?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.