microsoft / yams Goto Github PK

YAMS (Yet Another Microservices Solution) is a library that can be used to deploy and host microservices in the cloud (e.g. Azure) or on premises

License: Other

C# 99.94% Batchfile 0.06%

yams's Introduction

YAMS

YAMS (Yet Another Microservices Solution) is a library that can be used to deploy and host microservices on premises, in Azure, or on other cloud service platforms. It offers the following features:

Quick deployments of microservices to any target environment (~1 minute deployments to Azure).
Sharing infrastructure (multiple microservices can be deployed to the same on premises or cloud service).
Scaling microservices independently.
Versioning of microservices, quick updates, reverts, etc.
Support for Upgrade Domains to minimize (and potentially eliminate) application downtime during updates, including first-class support for Azure Upgrade Domains.
Microservices can be developed in any programming language and deployed with YAMS (as long as your service can be started with an exe).
Health monitoring and graceful shutdown of microservices.

YAMS has first-class support for deploying applications from Azure blob storage, but with its pluggable storage architecture, other providers such as SQL Server or file storage can be created and plugged in as well.

To deploy an application to a YAMS cluster, simply drop the binaries of the application into YAMS deployment storage. The binaries are then picked-up by YAMS, deployed to all VMs in the cluster, and then launched.

Please read the documentation below for more information.

Documentation

NuGets

Module	NuGet
Etg.Yams
Etg.Yams.Client
Etg.Yams.Powershell

Videos

Contribute!

We welcome contributions of all sorts including pull requests, suggestions, documentation, etc. Please feel free to open an issue to discuss any matter.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

License

This project is licensed under the MIT license.

yams's People

Contributors

Stargazers

Watchers

yams's Issues

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Yams studio can not be running because of the Newtonsoft.Json version binding setting in App.config

I tried to run the Yams studio, but exception occurred 👍
System.Windows.Markup.XamlParseException was unhandled
Message: An unhandled exception of type 'System.Windows.Markup.XamlParseException' occurred in PresentationFramework.dll
Additional information:

After I move the code in ctor for the main window, then I could see it is the issue of Newtonsoft.Json version conflicts.

Another thing is could you please show where to see the doc on how to use this lib?
And the design for this lib, I am really interested in this project.

Thanks

David

Extend client logs with app identity

Currently traces generated by Yams Client do not contain app identity.
It would be helpful to extent Yams Client to receive app identity from Yams Host so all the logs woudl contain app name / version.

Implementation suggestions from gitter:

Nehme Bilal @nehmebilal - Mar 31 17:52
await _process.Start($"{args} --HealthPipeName {_ipcConnection.ConnectionId}");
this is what the HealthProcessDecorator does
I would say we should probably modify the Process class to always append app name
you'll need to modify the YamsClientFactory as well to consume that argument
see YamsClientOptions

refactoring the YAMS model: from deployments to clusters and node roles

I think it'd be worth thinking about separating the Deployment (and DeploymentId) abstraction into separate abstractions for Cluster and NodeType (or role), making both of these ideas first-class citizens in the YAMS model.

Within a cluster, we sometimes want to designate different node types, installing different collections of services and apps on each type of node. But we also want to make sure we don't mix up deployments to different clusters/environments, keeping their definitions completely separate. I don't ever want one role in a cluster to affect software configuration on other clusters (accidentally or intentionally). If a second cluster has nodes with the same roles as nodes in another environment, it should be considered separate when any changes are rolled out. Running a tool to update cluster configurations should only affect components on one cluster at a time.

When fetching deployment configuration, a node would say "I belong to cluster x, and I'm node type y", and would then receive the appropriate deployment configuration object.

It's easy enough to fake this now, with the current state of YAMS, creating a single DeploymentId out of two parts like: "QAT-StorageNode", "QAT-ComputeNode", "Prod-StorageNode", "Prod-ComputeNode", etc. But it'd be nice for YAMS to recognize this reality with first-class recognition of both concepts, which are currently combined into a single "deployment" concept.

A current YAMS "deployment" would in this new model consist of all the AppPackageInstalls within a specified NodeType.

When I put my IT admin hat on, I want to see all the clusters, which node types are configured for each environment, and which packages are configured to run on each type of node.

With the right storage abstractions, that becomes much easier. A GUI tool with a TreeView could break down these three levels, let users expand a cluster environment node to see node types, expand those to see packages, drag packages into a node type for a particular cluster to add a reference to the package, etc.

I also imagine AppPackageInstalls having columns for ConfigFilesPackage, which would be just another .zip file (in our implementation) whose contents are unzipped into the app version directory with the rest of the app binaries. This would decouple file-based config (which is still very common and sometimes not within our control) from application binaries deployment, which is great because currently we'd need to publish a different "version" of the app package for it to be recognized with its updated config file. This is a waste of app packages (and deployment storage / disk space), which otherwise could have their versioned packages reused across several/many cluster environments, each with their own dynamic config.

Going along with ConfigFilesPackage, I want to add another property/column for CommandLineArguments (or ExecutionArguments), which would be a string of constant values and token expressions, to be replaced with values by YAMS when it executes the app/service.

Both would be optional.

@nehmebilal mentioned using some kind of extended properties dictionary or property bag to store things like ExecutionArguments, or other platform- or solution-specific parameters.

With this design, both cluster/node configurations and app/service packages could still be immutable. But in separating binaries from config, or providing an optional "config overlay of files", the packages of binaries can be reused across cluster environments, each with their own config files and/or command-line arguments.

Changes to IDeploymentRepository:

Task<ClusterConfig> FetchClusterConfig(string clusterId)
Task<NodeTypeConfig> FetchNodeTypeConfig(string clusterId, string nodeTypeId)
Task PublishClusterConfig(ClusterConfig clusterConfig)
Task PublishNodeTypeConfig(NodeTypeConfig nodeTypeConfig)
Task<bool> HasApplicationConfigFiles(AppIdentity appIdentity)

Downloading of application config files, if any exist, can be done within the implementation of IDeploymentRepository.DownloadApplicationBinaries, and a HasApplicationConfigFiles function isn't needed if that data is returned from within NodeTypeConfig.

ClusterConfig would be the ID plus a collection of NodeTypeConfig objects.

I think what I'm proposing is pretty generic. If many YAMS clusters out there will have only a single node type, that's fine. It can by default have a single node type, and all config actions naturally go against that. Once a second node type has been defined/created for a cluster, commands can be more specific about which node type they're assigning app packages to.

A NodeType is simply: a collection of versioned software packages to be installed on the same machine (virtual or otherwise).

AppInstallSetForNode would be a more descriptive name than NodeType, but is otherwise completely awful.

Thoughts?

Compress application binaries

At the moment, we upload / download application binaries file-by-file to / from blob storage. It would be good to have the option of uploading a compressed package that will then be downloaded by each VM and decompressed before starting the app.

As discussed below, we're also considering using NuGet for packaging and possibly as an alternative to blob storage.

Thanks to @galvesribeiro for the suggestion.

Add package version to assembly info

Currently you cannot identify Yams version used to build your app. This is info dll's build with Yams 1.6.1:

Please, make it synchronized with nuget package version :)

Is this project alive?

ITNOA

Hi,

As I can see this project have 50 open PR, and latest commits is for one year ago,

So My question: is this project alive?

thanks

Nuget package target framework

While trying to install Yams through nuget to a new worker role I'm getting the following message.

Could not install package 'Etg.Yams 1.0.13'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.5', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author

My project was initially targeting .net 4.5.1 but I lowered it when I got that error for the first time. Does it not support these versions or is something wrong with the package perhaps?

Edit --
Looks like it installs fine on framework 4.6.1.

Yams Client - confusion with arguments

Hello,

We ran into some problems with Yams Client arguments/options (pipe names). In addition to --ExitPipeName we also use some other options and arguments. If we pass all arguments into the YamsClientFactory app shuts down (Parser.Default.ParseArgumentsStrict(config.ProcessArgs, options) fails ==> and Environment.Exit(DefaultExitCodeFail) kills the app).

It is unclear how the argument parser works - which argument is which pipe, and/or option names are skipped/mapped to proper argument values.

As arguments are not very clear and straightforward, explicit properties in the newer versions for pipe names would probably be better?

Maybe something like
IYamsClient CreateYamsClient(string initializationPipeName, string exitPipeName, string healthPipeName, YamsClientConfig yamsClientConfig);

Very cool framework tho, we use it in production ;)

Kind regards,
Heko

Minimize network traffic during app deployment

YAMS can be smarter when it comes to downloading app binaries. We can make sure that the apps are downloaded once only and even copy files from previous, locally stored version if the file MD5 hash matches.

Reuben's Yams Uploader is a good starting point:
https://github.com/ReubenBond/YamsDeploy/blob/master/YamsApplicationManager.cs

DI make CloudBlobClient a singleton

This is just a minor optimization in DI: the CloudBlobClient can be most likely made a singleton.

System.IO.IOException: There is not enough space on the disk.

I have a Cloud Service of medium with 2 instances running Yams host. There are 4 microservices deployed with sizes of 15MB, 16MB, 14MB and 14MB.

During upgrades I'm getting Failed to perform update; Exception was: System.IO.IOException: There is not enough space on the disk.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath) 
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) 
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access) 
at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.BeginDownloadToFile(String path, FileMode mode, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, AsyncCallback callback, Object state) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 405 
at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.BeginDownloadToFile(String path, FileMode mode, AsyncCallback callback, Object state) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 386 
at Microsoft.WindowsAzure.Storage.Core.Util.AsyncExtensions.TaskFromVoidApm[T1,T2](Func`5 beginMethod, Action`1 endMethod, T1 arg1, T2 arg2, CancellationToken cancellationToken) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Util\AsyncExtensions.cs:line 190 
at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadToFileAsync(String path, FileMode mode, CancellationToken cancellationToken) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 511 
at Microsoft.WindowsAzure.Storage.Blob.CloudBlob.DownloadToFileAsync(String path, FileMode mode) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlob.cs:line 498 
at Etg.Yams.Azure.Utils.BlobUtils.<DownloadBlobs>d__2.MoveNext() 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
at Etg.Yams.Azure.Storage.BlobStorageDeploymentRepository.<DownloadApplicationBinaries>d__8.MoveNext() 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) 
at Etg.Yams.Download.ApplicationDownloader.<DownloadApplication>d__3.MoveNext() 
--- End of stack trace from previous location where exception was thrown --- 
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Etg.Yams.Update.ApplicationUpdateManager.<CheckForUpdates>d__6.MoveNext()

There is IMHO more than enough free space on all drives:
C: (Yams) - 481GB
D: (OS) - 20GB
E: - 780MB

When the error occurs the local folders for the latest versions of all 4 microservices contain only some of the files.

What could be the cause? How can I diagnose the issue further?

Use XUnit for testing

@galvesribeiro suggested that we move our tests to XUnit which has better support for .Net Core and cross platform development.

@galvesribeiro Also volunteered to make the change :)

Tracking this here.

REST API

I would like to expose current cluster configuration / application status through a REST API. This will enable Load Balancer integration through HTTP probes and allow for future tooling.

The API can expose following information:

Cluster membership - list of nodes
Individual node status (uptime / heartbeat)
Application configuration
Current status of deployed application on each node
Health information from each application

We can extend current health monitoring to allow additional custom data to be exposed from the application.

The above will require adding a self-hosted HTTP server to be integrated with the YAMS host and also a simple implementation of cluster membership so all nodes in the cluser can be identified.

Tutorial for on premise needs the cloud?

The tutorials read, like I can run it on premise, but need a azure storage anyway?

See: https://github.com/Microsoft/Yams/blob/master/Samples/Etg.Yams.ARM/README.md

storageAccount: Azure Storage account name

storageKey: Azure Storage account key

Support prerelease versions like in SemVer

I would like to be able to deploy prerelease versions of an app, that have same version part and some suffix, like:

0.9.0-feature1
0.9.0-feature2
0.9.0-bugfix

Monitor Application health

When an application is deployed, each Yams instance downloads the application from storage and start a process using the provided exe. Yams then assumes that the process has started correctly and moves on to the next app.

It would be great if Yams can make sure that the app has started correctly. It would be also great if the current behavior is preserved if the app itself has no support for health checks to make sure that we can still run any exe in Yams. We still need to define how Yams and the app will communicate.

In particular, during an upgrade of an application, Yams should rollback to the previous working version if the new version fails to start. Yams should not move to the next fault / upgrade domain until the current one has been successfully upgraded (or rolled back).

Thanks to @ReubenBond for the suggestion.

Yams crashes with broken pipe

When using health features of YAMS that involve the use of named pipes, killing an app cause the named pipe to break and cause the whole YAMS host to crash.

Topshelf v4.0.0.0, .NET Framework v4.0.30319.42000
The Yams service is now running, press Control+C to exit.
Topshelf.Hosts.ConsoleRunHost Critical: 0 : The service threw an unhandled exception, System.AggregateException: One or more errors occurred. ---> System.IO.IOException: Pipe is broken.
   at System.IO.Pipes.PipeStream.CheckWriteOperations()
   at System.IO.Pipes.PipeStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Etg.Yams.Process.Ipc.IpcConnection.Disconnect()
   at Etg.Yams.Process.Ipc.IpcConnection.Dispose()
   at Etg.Yams.Process.GracefulShutdownProcessDecorator.Dispose()
   at Etg.Yams.Process.AbstractProcessDecorator.Dispose()
   at Etg.Yams.Process.MonitorInitProcessDecorator.Dispose()
   at Etg.Yams.Process.SelfRestartingProcess.Dispose()
   at Etg.Yams.Application.ConfigurableApplication.Dispose()
   at Etg.Yams.Application.ApplicationPool.<RemoveApplication>d__7.MoveNext()
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Etg.Yams.Application.ApplicationPool.OnApplicationExited(Object sender, ApplicationExitedArgs args)
   at Etg.Yams.Application.Application.OnProcessExited(Object sender, ProcessExitedArgs e)
   at Etg.Yams.Process.SelfRestartingProcess.OnFailed(String format, Object[] args)
   at Etg.Yams.Process.SelfRestartingProcess.<ExitedTryRestart>d__18.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c.<ThrowAsync>b__6_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
---> (Inner Exception #0) System.IO.IOException: Pipe is broken.
   at System.IO.Pipes.PipeStream.CheckWriteOperations()
   at System.IO.Pipes.PipeStream.Flush()
   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
   at System.IO.StreamWriter.Dispose(Boolean disposing)
   at System.IO.TextWriter.Dispose()
   at Etg.Yams.Process.Ipc.IpcConnection.Disconnect()
   at Etg.Yams.Process.Ipc.IpcConnection.Dispose()
   at Etg.Yams.Process.GracefulShutdownProcessDecorator.Dispose()
   at Etg.Yams.Process.AbstractProcessDecorator.Dispose()
   at Etg.Yams.Process.MonitorInitProcessDecorator.Dispose()
   at Etg.Yams.Process.SelfRestartingProcess.Dispose()
   at Etg.Yams.Application.ConfigurableApplication.Dispose()
   at Etg.Yams.Application.ApplicationPool.<RemoveApplication>d__7.MoveNext()<---

Stopping the Yams service
The Yams service has stopped.

https://stackoverflow.com/questions/45308306/gracefully-closing-a-named-pipe-and-disposing-of-streams

YAMS should handle this exception and only the app that has been killed should be affected, not the whole YAMS instance.

Consider creating separate Host and Client NuGet packages

The new advanced client features (monitored deployments, health checks, graceful shutdown) require yams applications to reference YAMS NuGet package to use YAMS client.

Currently YAMS NuGet package contains several dependencies that are not needed by the client, 'polluting' package dependencied of the client application. This also increases the risk of conflicts between versions required by YAMS and the application.

What is more, Client NuGet package will probably evolve slover than the host so it can be versioned separatelly.

I think a separate NuGet packages for Host and Client will minimize the impact on using advanced YAMS features in individual applications.

Not working via Emulator

Running host via VS administrator.
Getting this error from yamsWOrker:

WaWorkerHost.exe] YamsWorker is running
[WaWorkerHost.exe] DeploymentId is deployment27(88)
[WaWorkerHost.exe] Checking for updates
[WaWorkerHost.exe] Failed to perform update; Exception was: System.UnauthorizedAccessException: Access to the path 'Microsoft.Owin.dll' is denied.
   at System.IO.Directory.DeleteHelper(String fullPath, String userPath, Boolean recursive, Boolean throwOnTopLevelDirectoryNotFound)

web app itself is being run as separate console application and only if I specify different ports for Owin host and service http endpoint.

Support for load balancing

This issue encompases the work necessary for the YAMS host to support load balancing during application deployment.

The current behaviour is as follows: once the configuration change is detected YAMS host will process each affected application in parallel, updating one Update Domain of each application at a time. There is no synchronization between applications so it is likely that while App 1 is being updated in Update Domain 1 at the same time App 2 is being updated in Update Domain 0. This guarantees the fastes possible deployment of all applications but also means that many/all Update Domains may be affected at the same time.

Currently there is no support for Load Balancing so while App 1 is being updated in Update Domain 1 the nodes in this UD are considered online and will continue to receive requests.

The desired behaviour is for YAMS host to notify Load Balancer that a certain Update Domain is being updated and take the relevant nodes offline so Load Balancer can route traffic to remaining nodes in different Update Domains. In case of Azure LB this can be achieved by implementing custom probe (https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-custom-probe-overview).

There is however one issue that may affect cluster availability: with the current deployment startegy that allows multiple Update Domain to be updated at the same time there is a risk that multiple / all Update Domains are actually brought offline. In order to aviod it I propose the following deployment starategy is implemented:

Nodes in Update Domain 0 are taken offline from Load Balancer
YAMS host upgrades all applications in Update Domain 0
If applications support monitored deployment feature the host waits for the application initialization to complete
Nodes in Update Domain 0 are brought online
Steps 1-4 are repeated for remaining Update Domains one at the time

The above strategy guarantees that only nodes in one Update Domain are offline at the time.
The only side effect is that in case of the hosting multiple applications with different deployment times the time of updating whole cluster is deployment time of the slowest application x number of Update Domains.

I would like to receive feedback from anyone who would like to keep the exising deployment startegy (not using Load Balancer and allowing traffic to be send to unavailable nodes). If anyone is interested we may keep supporting it but as this will require additional work some convincing will be required.

The design of the solution

A new interface ILoadBalancer will be introduced.

public interface ILoadBalancer
{
    Task TakeOffline();
    Task BringOnline();
}

Two implementations of ILoadBalancer will be provided: NoLoadBalancer and AzureLoadBalancer.
The former will return completed tasks and can be used if load balancing is not needed. The latter will use custom probes to notify Azure Load Balancer of node availability.

The ApplicationInstaller will be extended to take ILoadBalancer as a dependency.

Make DeploymentConfig serializable to JSON

It would be great if this kind of JSON serialization could work with DeploymentConfig and its supporting types.

var json = JsonConvert.SerializeObject(deploymentConfig);