pulumi / pulumi

Pulumi - Infrastructure as Code in any programming language. Build infrastructure intuitively on any cloud using familiar languages 🚀

Pulumi's Infrastructure as Code SDK is the easiest way to build and deploy infrastructure, of any architecture and on any cloud, using programming languages that you already know and love. Code and ship infrastructure faster with your favorite languages and tools, and embed IaC anywhere with Automation API.

Simply write code in your favorite language and Pulumi automatically provisions and manages your resources on AWS, Azure, Google Cloud Platform, Kubernetes, and 120+ providers using an infrastructure-as-code approach. Skip the YAML, and use standard language features like loops, functions, classes, and package management that you already know and love.

For example, create three web servers:

const aws = require("@pulumi/aws");
const sg = new aws.ec2.SecurityGroup("web-sg", {
    ingress: [{ protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] }],
});
for (let i = 0; i < 3; i++) {
    new aws.ec2.Instance(`web-${i}`, {
        ami: "ami-7172b611",
        instanceType: "t2.micro",
        vpcSecurityGroupIds: [sg.id],
        userData: `#!/bin/bash
            echo "Hello, World!" > index.html
            nohup python -m SimpleHTTPServer 80 &`,
    });
}

Or a simple serverless timer that archives Hacker News every day at 8:30AM:

const aws = require("@pulumi/aws");

const snapshots = new aws.dynamodb.Table("snapshots", {
    attributes: [{ name: "id", type: "S", }],
    hashKey: "id", billingMode: "PAY_PER_REQUEST",
});

aws.cloudwatch.onSchedule("daily-yc-snapshot", "cron(30 8 * * ? *)", () => {
    require("https").get("https://news.ycombinator.com", res => {
        let content = "";
        res.setEncoding("utf8");
        res.on("data", chunk => content += chunk);
        res.on("end", () => new aws.sdk.DynamoDB.DocumentClient().put({
            TableName: snapshots.name.get(),
            Item: { date: Date.now(), content },
        }).promise());
    }).end();
});

Many examples are available spanning containers, serverless, and infrastructure in pulumi/examples.

Pulumi is open source under the Apache 2.0 license, supports many languages and clouds, and is easy to extend. This repo contains the pulumi CLI, language SDKs, and core Pulumi engine, and individual libraries are in their own repos.

Welcome

• Get Started with Pulumi: Deploy a simple application in AWS, Azure, Google Cloud, or Kubernetes using Pulumi.

• Learn: Follow Pulumi learning pathways to learn best practices and architectural patterns through authentic examples.

• Examples: Browse several examples across many languages, clouds, and scenarios including containers, serverless, and infrastructure.

• Docs: Learn about Pulumi concepts, follow user-guides, and consult the reference documentation.

• Registry: Find the Pulumi Package with the resources you need. Install the package directly into your project, browse the API documentation, and start building.

• Pulumi Roadmap: Review the planned work for the upcoming quarter and a selected backlog of issues that are on our mind but not yet scheduled.

• Community Slack: Join us in Pulumi Community Slack. All conversations and questions are welcome.

• GitHub Discussions: Ask questions or share what you're building with Pulumi.

Getting Started

See the Get Started guide to quickly get started with Pulumi on your platform and cloud of choice.

Otherwise, the following steps demonstrate how to deploy your first Pulumi program, using AWS Serverless Lambdas, in minutes:

1. Install:

   To install the latest Pulumi release, run the following (see full installation instructions for additional installation options):

   $ curl -fsSL https://get.pulumi.com/ | sh

2. Create a Project:

   After installing, you can get started with the pulumi new command:

   $ mkdir pulumi-demo && cd pulumi-demo
   $ pulumi new hello-aws-javascript

   The new command offers templates for all languages and clouds. Run it without an argument and it'll prompt you with available projects. This command created an AWS Serverless Lambda project written in JavaScript.

3. Deploy to the Cloud:

   Run pulumi up to get your code to the cloud:

   $ pulumi up

   This makes all cloud resources needed to run your code. Simply make edits to your project, and subsequent pulumi ups will compute the minimal diff to deploy your changes.

4. Use Your Program:

   Now that your code is deployed, you can interact with it. In the above example, we can curl the endpoint:

   $ curl $(pulumi stack output url)

5. Access the Logs:

   If you're using containers or functions, Pulumi's unified logging command will show all of your logs:

   $ pulumi logs -f

6. Destroy your Resources:

   After you're done, you can remove all resources created by your program:

   $ pulumi destroy -y

To learn more, head over to pulumi.com for much more information, including tutorials, examples, and details of the core Pulumi CLI and programming model concepts.

Platform

Languages

	Language	Status	Runtime	Versions
	JavaScript	Stable	Node.js	Current, Active and Maintenance LTS versions
	TypeScript	Stable	Node.js	Current, Active and Maintenance LTS versions
	Python	Stable	Python	Supported versions
	Go	Stable	Go	Supported versions
	.NET (C#/F#/VB.NET)	Stable	.NET	Supported versions
	Java	Public Preview	JDK	11+
	YAML	Stable	n/a	n/a

EOL Releases

The Pulumi CLI v1 and v2 are no longer supported. If you are not yet running v3, please consider migrating to v3 to continue getting the latest and greatest Pulumi has to offer! 💪

• To migrate from v2 to v3, please see our v3 Migration Guide.

Clouds

Visit the Registry for the full list of supported cloud and infrastructure providers.

Contributing

Visit CONTRIBUTING.md for information on building Pulumi from source or contributing improvements.

pulumi's Issues

Project the full set of AWS resource types

At the moment, we're only projecting the bare minimum AWS resource types needed to get various proofs of concepts, demos, and the like, up and running. Eventually we want to project the full set.

Nag mails service

We should have a feature in our service that sends nag mails for required configuration updates (e.g., stale dependencies). This is particularly nice if Pulumi can pre-create and validate the change, enabling a bot to even create a PR for you. This might include escalation policies, for instance, disabling a service if a critical security updated hasn't been addressed within a certain timeframe.

Real module names

At the moment, we just carry forward the ECMA/TypeScript notion that module names are simply their file names. That leads to absolute paths in our baselines and generally unfriendly Mu names that cannot be bound according to our dependency resolution logic. This should be a short-lived limitation, however this bug tracks fixing this so that we don't forget.

Specify and implement MuGL

Mu Graph Language (MuGL) is a very simple graph description markup language. Please see https://github.com/marapongo/mu/blob/master/docs/design/formats.md#mu-graph-language-mugl for high-level details on it. As soon as we can lower to an AST, we will need to specify and implement this format, so that the AST can be evaluated to generate a MuGL graph.

Create an OSS roadmap

We need a roadmap, and overall plan, for OSS'ing. Most likely we will roll this up underneath some combination of customer success, initial version of a service, plus company formation and funding, in order to create sufficient momentum to attract attention (a "splash", if you will).

Initial MuJS implementation

This tracks implementing the initial MuJS compiler. This must specify a well-defined subset of JavaScript that is amenable to MuPack/MuIL targeting. It must then lower that subset to the MuPack/MuIL format/AST, and emit self-describing metadata, so that the Mu toolchain can read it, bind the elements as-is, and evaluate it to generate MuGL (#39).

Design secrets

The secrets system needs to be designed. In particular, we want secrets to be opaque cookies that can flow throughout the system without concern for their safety. At the time they are wielded and redeemed for the concrete secret value -- as close to their use as possible -- this opaque cookie is turned into an authority, perhaps through some combination with an authorized identity.

Implement Mu's initial cross-cloud abstractions

The mu/x namespace will house a set of "logical" cloud-agnostic types that internally map to cloud-specific resources; e.g., mu/x/table, mu/x/queue, mu/x/topic, mu/x/volume, and so on. A (slightly out of date) braindump of what this might look like can be found here.

More about this topic can be found in the Cross-Cloud Targeting document. As part of this work item, we should flesh that document out in greater detail.

Support ECMAScript destructuring in MuJS

To scope the initial MuJS compiler, we have intentionally omitted support for destructuring. It isn't supported in MuPack/MuIL, and requires fairly sophisticated rewrites within the compiler. For now, we have assumed identifiers in most places, which is a gross simplification. After M0, we'll probably want to return to this, especially since I expect us to want to use it for stack constructors.

Configuration (and secret) design

At the moment, we lack a configuration and secret subsystem.

Consider consolidating project files

In a93841d#commitcomment-20454804, we discussed consolidating the various project files.

For example, a JS package has at least two: *.yaml|json and tsconfig.json. It would seem much nicer to just have one. We could put the Mu package information in tsconfig.json since TypeScript ignores "excess" metadata; or, we could just put the TypeScript options and files into *.yaml|json.

I personally prefer hijacking *.yaml|json, for three reasons: 1) having a *.yaml|json file is a nice "marker" that a given directory/repo contains a cloud package at a quick glance (similar to Dockerfile); 2) I always worry about hijacking existing things like file formats because then people Google for them and get confused because what they are seeing doesn't line up with documentation about that thing; and 3) it's not a given that all languages and package formats will ignore excess.

But there are advantages to tsconfig.json too, namely, we can use the vanilla TypeScript compiler in addition to the JS compiler to compile the project.

If we went down the *.yaml|json path, for example, we might find *.yaml:

name: my/stack/pack
description: A great stack pack!
language: javascript
compilerOptions:
    target: es6
files:
    - lib/diag/index.ts
    - and so on

It gets more complex, however, when it comes to package management. A project may or may not also have a third file, package.json, depending on where we land on package management. I think for the MVP, we will be doing Git-based package management on our own; however eventually we will need to think deeply about the interaction between meta-languages and their host languages being intermixed, in which imports need to be resolved (see #47).

Finally, there is an argument against doing all of this. Each file caters to distinct concerns. *.yaml|json contains the cloud-specific metadata; tsconfig.json contains the information required to compile a project; and package.json contains just the package management metadata. Each tool processes a different file and each file contains only what is required for that tool. This means, for instance, that we can omit the tsconfig.json entirely from a Node.js package because we will have precompiled it before publishing, such that clients needn't care how we built it.

Expand templates during compilation

At the moment, we don't expand templates during compilation. We need to.

The intent is to use Go text templates, because it has all of the expressiveness we want out of a templating language. (Conditionals, custom functions, etc.) An alternative is to use something like Jsonnet, however (a) it is probably less familiar to the population at large, and (b) we want something that is file format agnostic.

As part of this, we need to decide what goes into the "context" object available to templates.

Code binding

MetaMus are configuration languages. But sometimes a MetaMu program describes a resource that must be bound to a real programming language artifact. This could be an artifact written in the MetaMu's true language alongside the configuration, or a piece of code sitting on disk.

For example:

• A container resource must bind to a Dockerfile plus the assets needed to build the image.
• A serverless MetaMu might have APIs and/or Lambdas expressed in code alongside the configuration. This code must be "shredded" into on-disk assets that are bound to.

In all cases, there is a binding layer that needs to be invented, for binding names to assets. There is also a package manager component of this, in that there must be a convention for where these assets reside so they can be published, downloaded, and bound to, at various places.

Lock down everything by default

We wish to ensure Pulumi is the only way to modify environments by default. As a result, step #1 of adopting it in an existing environment should be to modify permissions to disable resource modification for any account other than Pulumi. Similarly, we probably want to disable direct SSH access to server resources. In both cases, it would be nice if any "exceptions" (like temporary SSH access during debugging) were supported by official and auditable workflows, including checking for drift afterwards to ensure our state still reflects live state, and vice versa.

Design for stacks exporting APIs

At the moment, we have placeholders for stacks being able to export structured APIs. This will enable stronger typechecking, including proper substitutability that our unique approach to componentization enables. The problem is, we don't yet have any idea of what form this takes.

At a top-level, we must pick a metadata format. Candidates here include OpenAPI/Swagger, RAML, and Protobufs/gRPC. One option is to pick something that is "provider"-like and pluggable.

I have to admit, I have a preference for OpenAPI/Swagger for "internet-facing" services, and Protobufs/gRPC for inward-facing microservices APIs.

After that, we need to figure out what runtime manifestation occurs on the wire. For instance, it's possible we would leverage OpenAPI/Swagger as the metadata format, but turn around and use gRPC and HTTP/2 on the wire.

Moreover, when installing a stack locally, we may want to auto-generate some client side proxy wrappers for interacting with the service, to add strong typing. This becomes more important as we consider doing the DSL.

This work item will likely spawn many more as we make progress on these topics.

mu apply: Implement deployments (for AWS/ECS)

As soon as building is limping along, we will want to package up the results, and deploy them to the target. At a top-level, the mu apply command will apply a set of changes to a target.

However, the following components are necessary before that can work:

• Any build metadata must be output to the right location.

• Any code artifacts must be packaged alongside the metadata outputs. The goal here is that a user never needs to manually zip stuff up, manually upload things to S3 (in the case of AWS), etc.

• A changeset "diff" must be generated, so that we can change only what's new and/or has been been updated or deleted. Note that this somehow needs to interact with the notion that some updates may need to recreate resources. Presumably this requires a flag to mu apply, such as mu apply --confirm-deletions. Also, either mu apply --dry-run, mu diff, or both, should work.

For the time being, I am thinking we will defer to CloudFormation for the heavy lifting. It is a very big open question whether we want to do this into the future, given the possibility of a lackluster user experience, less control over the mechanics, and less visibility into auditing/logging what goes on.

Emulate pre-arrow "this" capture in MuJS

At the moment, we use normal "this" capture rules in MuJS. This isn't correct for old-style callbacks that don't use arrow functions. For those, we will need to emulate the legacy "this" context.

Resource providers

The resource provider model is how new "primitive" resource types are added. This will work by annotating programs in a certain way (as an intrinsic in IL), and then having a dynamic plug-in load and execute CRUD commands as needed in response to graph analysis and traversal.

Design for local stack execution

One of the major appeals for Pulumi is the ability to run stacks locally. We need to figure out how exactly that works, especially for scheduler-less targets (i.e., no Docker Swarm or Kubernetes).

Create a design/specification for Mufiles

Explore a new approach to templating

I'm increasingly nervous about having a full blown templating language, such as Go's text/template system, for Mufiles. For a number of reasons:

• It is difficult to semantically analyze. As we start doing the Hub, this will be problematic.

• It isn't amenable to checking. Ideally when you build a file, you would know that all paths are valid markup. Ideally you'd be able to warn about invalid template expansions. To do that would require a templating language that is integrated into the semantics of the enclosing markup kind.

• Related to this, the failure modes during template expansion at compile-time (see #7) are actually quite terrible. For example, if anything in your transitive closure doesn't happen to produce valid YAML during template expansion, you're left debugging it, much like a C++ template failure.

• It can introduce non-determinism into the results.

I've done some thinking and feel like we could probably get away with a drastically reduced set of operators that are cleanly integrated with the markup semantics. Furthermore, with integration with our schema work (see #9), I feel like better compile-time checking could be applied to them, without resorting to arbitrary Turing complete code. For example, we would just support:

• {{.property}}: property expansion.
• {{if .property}}...{{fi}}, {{else if .property}}, {{else}}: property-driven conditionals.
• {{include "file.yaml"}}: template inclusion.
• {{error "Message" arguments}}: custom template validation.
• {{require .property "Message" arguments"}}: property existence validation.

Maybe a handful of others would appear but this is a good start.

I'll keep going with Go templates for now, since they can express a superset of this and hence are sufficient for creating a prototype, however we will want to do this before releasing anything publicly. I'm also internally conflicted about the fact that DSL-driven files will bring about many of the above-mentioned challenges, which kind of stinks, since I think the DSL will be great for usability.

Emit dependencies and proper type names

At the moment, all of our conversions are "syntactic"; that is, with few exceptions, we only rely on the parse tree information to generate the MuPack/MuIL tree. This means we don't emit bound types from TypeScript's semantic analysis tree, hence some random "TODO" types in the output.

We need to start emitting fully bound information. This is more complex than it seems, because we not only need to consult the TypeChecker -- versus relying exclusively on the AST nodes as we are doing now -- and, perhaps more difficult, we must lower the type names to proper MuIL symbol tokens. These must be references as described in https://github.com/marapongo/mu/blob/master/docs/design/deps.md. As part of that, we will need to also emit the dependencies.

Emitting dependencies will also be difficult, since all dependencies in TypeScript are source-based, and so this information is "lost" by the time we are compiling. We may need to recover it from the package metadata in the locations TypeScript has picked up certain definition files from.

Implement AWS resource providers

We want a full projection of the AWS resource surface area as resources. This entails schematizing them probably figuring out things like the configuration subsystem (for AWS credentials).

Design for AWS IaaS targeting

We need a design for how to target AWS as the cloud provider and ECS as the scheduler.

Decide whether to allow cycles

There are circumstances in which the requirement that service graphs are acyclic limits legitimate use cases. For example, consider this case: all services in a topology must log; that logging must be persisted; yet, the persistence service is a service like all others, and so it must log.

At the moment, there are a few ways out of this Catch-22:

• Bundle the logging and persistence service together.
• Use a dynamic dependency for the logging service's dependency on persistence.
• Clone the persistence service used by logging, and simply omit logging.
• Use an intrinsic in the system to model this relationship.

None of these are great. But clearly something must be "different" about this relationship already, to avoid infinite recursion at runtime, so arguably this snag is a welcomed one.

One possibility is to permit co-creation of services. This would require provisioning of graphs in passes. Another possibility is to allow mutable properties on services, such that, for example, in the above case the persistence dependency could be added to logging after it has been created.

We feel that acyclic graphs are important to the overall static analyzability of the system, and the ability to incrementally apply changes in true immutable infrastructure fashion, so we will hold the line. This work item tracks the issue, however, as it will surely come up again.

Code bindings for serverless code and more

A code binding concept should make it possible to transform a code asset expressed at the MetaMu layer into a logical symbol in MetaIL, and then to bind that symbol to a physical code asset in the respective resource provider. For this, I am envisioning a new CodeAsset intrinsic type.

To illustrate why we want this, consider how you would express an AWS Lambda in your favorite MetaMu. There will be a function written in code separate from the configuration language; we will then zip up that code; describe the code to configuration in terms of dynamic entrypoints; transfer the zip file to an S3 bucket; reference that bucket in the configuration; and then (hopefully) clean up that S3 bucket afterwards (or not, as is commonly the case, due to the race conditions involved).

This can be mapped to stack types in Mu simply by exposing the raw properties as-is. Of course, since those are what AWS deals with, we must translate whatever a developer writes into them. But I envision this higher level CodeAsset type enabling a simpler expression of all of this.

The real motivating examples come when you look at MetaMu coexisting side-by-side with their regular language counterparts. For example, in the lambda case, perhaps we write code like:

let func = new aws.Lambda((ctx) => {
    // lambda code goes here
});

In this example, we can pass a true lambda as the argument to a Lambda stack. This is typed as CodeAsset and the MetaMu compiler knows how to prepare the lambda as a CodeAsset -- carrying the physical package's location on disk, entrypoint information, and necessary runtime (in this case, NodeJS version X) -- and then the resource provider will translate this intermediate CodeAsset information into the properties required by AWS Lambda (zipping, S3 bucketing, etc).

There are other examples. For instance, we probably want something similar for an API gateway:

let website = new aws.WebSite({
    "/": (req, res) => { /* handle "GET /" */ },
    "/customer", (req, res) => { /* handle "GET /customer" */ },
    ... etc ...
});

In this case, the WebSite constructor is just taking an array of CodeAssets. Note that because these types add some "artistic license" atop the raw AWS abstractions, they probably would exist in a separate package devoted to handling common AWS coding patterns.

As yet another example, imagine we have an RPC server. We might even envision nicer programming models than callbacks for this, perhaps using decorators:

class MyServer extends mu.x.RPC {
    @api public async register(customer: Customer): Promise<void> {
        // registration code goes here
    }
}

In this example, we can imagine that @api is a decorator that derives from @mu.codeasset; this directs the MetaMu compiler to handle it must like a lambda binding to a CodeAsset, enabling the resource provider for the mu.x.RPC type to grovel all public members, and bind them to API endpoints. This could be just a simple way of defining an aws.WebSite, or perhaps it could even lead to a pluggable RPC server which uses gRPC or Thrift.

Figure out graph queryability

A scenario to consider is queryability of resource graphs. This covers multiple scenarios:

• Simple questions about the contents of a graph ("What versions of Ubuntu am I running?")
• Complex questions that combine static analysis ("What VMs are vulnerability to CVE-Xxx?")
• Registering event notifications off the graph ("Tell me if any of my nodes ever get out of date")
• Even making modifications to the graph based on queries ("Patch all old Ubuntu versions")

Furthermore, this could be exposed in a bot-like interface (text or speech), for real bot-driven IT/configuration management workflows.

There are many graph technologies to consider, however we should keep this in mind as we design.

Implement property binding/typechecking

Auto-increment build numbers

At the moment, the Lumi toolchain's version is hard-coded as 0.0.1.

Ideally, we would manually bump the major and minor versions, but have the revision auto-increment or base itself off the timestamp/date/hash/something of the build.

Decide how serious we are about capabilities

At the moment we claim to use capabilities to represent service dependencies.

No matter what, these dependencies are present in the metadata, so it's not entirely a lie. But in the programming model, thanks to the presence of mutable global state, this intended explicit model can be circumvented; e.g., imagine stashing the logging service in some mutable global map, where it can be discovered by anyone, including possibly third party libraries.

This might not be a bad thing. The model is encouraging a "capability-like" model, and retaining everything it needs in the metadata, but does not prohibit abuse or the creation of new patterns we cannot foresee now (like a global context object). But it does feel like a middle ground.

We need to decide how we feel about this. We might, for instance, wish to embellish the notion of a global context object with a bag of services, if that becomes a common way to use the system.

We might, on the other hand, decide to go all-in on explicit capabilities and plug the mutable global state hole. It could be that we need to support object freezing anyway due to JavaScript. In that world, we could envision tagging all objects created within global initializers as frozen. This would unfortunately lead to the possibility of more runtime failures, so isn't wholly attractive to me.

Platform implementation roadmap

The following is a platform implementation roadmap for IaaS/CaaS combinations. The lower the number, the higher the priority. * means in progress. Nonsense combinations are left blank:

	AWS	GCP	Azure	VMWare
none (VMs)	1*	5	10	200
Docker Swarm	3	50	101	201
Kubernetes	4	7	102	202
Mesos	400	300	103	203
ECS	2
GKE		6
ACS			11

Note that the "none (VMs)" row is essentially the "base case" for each IaaS provider, hence for anything in its column, we will tackle that row first.

I will keep this up-to-date as we go. All targets are described in more detail inside the metadata format specification.

Design package manager protocols

We have decided to adopt a mixture of NPM, Golang, and Docker Hub-like package management styles. However, most likely we will carry one aspect of Golang forward, which is the way it seamlessly consumes Git repos as packages (see https://github.com/marapongo/mu/blob/master/docs/deps.md).

We need to fully specify this protocol so that we can implement it.

Implement Mu's primitive types for "aws:ecs"

We need to implement the full set of mu/* types for the "aws:ecs" cloud/scheduler provider pair.

Decide whether to pursue Mull and/or verification

As we began reaching the limits of templating and JSON Schema-like typechecking, we sketched out a possible "integrated" approach, Mull, that amounts to a full-blown language extension to YAML and JSON. See here for the spec: https://github.com/marapongo/mu/blob/master/docs/language.md.

Since then, I've begun pursuing the DSL architecture (#22). That will give us the typechecking plus "templating" that we desire. But that is likely to compile down into metadata of some form (an IL, if you will), and it remains to be seen whether we will want to typecheck/verify that. Most likely, in the limit of time, yes, but perhaps not in the short-term.

This work item tracks deciding on Mull and verification overall.

Strict JSON/YAML unmarshaling

At the moment, Go's JSON marshaler -- and the YAML parser which mimics it -- ignores unrecognized properties. That is, properties without field mappings. This is problematic for two reasons:

1. We'd prefer to issue errors/warnings for this case, to help users diagnose subtle issues, like the wrong casing, placing a property in the wrong place, etc.

2. In a few cases, we actually want to preserve those properties. For example, services have a set of common "base" properties, but are generally extensible for different service types.

A number of changes are accumulating for an upcoming release. I've got my eye on golang/go#6213, as it seems to address both of these at once. If that comes along fast enough, great, otherwise we may need to bite off doing this manually. (And in fact that may be partly inevitable because I doubt the 3rd party YAML parser we're using will catch up that fast.)

Related to this, but slightly different, we also want to issue errors during parsing for missing required properties. We check for these manually right now but it'd be much nicer to do it declaratively and let the marshaler take care of this. It's unclear if all of the pending Go changes for the JSON parser will support something like this.

Implement rudimentary package management commands

As soon as #5 is completed (the package manager protocol design), we should implement the basic package management commands (namely mu get and, ideally, mu list).

Switch back to official YAML repo

The ghodss/yaml package contains a bug that I tripped on when converting our AST data structures over to using pointers in their maps instead of values. (This was required to avoid making copies of nodes which caused problems elsewhere.) Unfortunately, this bug seems to be known: there is an outstanding PR ghodss/yaml#7 and at least one open issue pertaining to this.

For now, I have forked the library to joeduffy/yaml, and fixed the bug. This is obviously not a great long-term strategy. I also didn't work to narrow down the root cause to a minimal repo, since I really don't understand much of why that reflection fu is in there to begin with.

I was tempted to navigate away from this library and towards go-yaml/yaml, however, then I read the original motivation for ghodss's approach: http://ghodss.com/2014/the-right-way-to-handle-yaml-in-golang/. I agree with the overall spirit (though worry about the performance). And even a 1 minute thought exercise of the annotation bloat that would result made my stomach ache.

So for now, I've gone the quick-and-dirty route. We likely need to revisit our choices here anyway as soon as we tackle #4, so I didn't want to invest too much time right now.

Detect cycles where they are problematic

In general, we've been "undecided" on cycles in a number of places.

However, it's almost certain that module imports need to be cycle-free. And most likely the dependencies among services expressed within projects.

This work item tracks detecting these cycles and issuing errors (versus what will happen now: infinite recursion, failure very late during compilation (e.g., CloudFormation-induced), or worse).

Design the DSL architecture

Mufiles are currently either JSON or YAML files. We will eventually permit code-driven stacks, however, using a pluggable DSL compiler model. This allows stacks to be written more concisely, especially when it comes to serverless scenarios. But the design here gets quite involved.

For example, rather than needing to call out every mu/func as an independent entity in the metadata, with a loose connection to the code, we can simply say things like this:

var mu = require("mu");
var mongo = require("mu/mongodb");
// this runs at design-time.
var db = new mongo.MongoDB();
db.changes.forEach(() => {
    // this runs at run-time.
});

In this example, we are creating a mongodb/mongodb service, an anonymous mu/func containing the lambda code, and then subscribing that to a mu/event named changes exposed by the mongodb/mongodb service, that triggers the mu/func anytime a change is made to the database.

I have a vague idea that running code like this happens "two ways":

1. In one mode, the mu APIs simply keep track of the objects constructed; as a result, at the end, they can produce the equivalent of the Mu metadata file. This is then applied in the usual ways.

2. In another mode, the mu APIs actually do things. This is the mode a deployed application would run in. There are two variants:

   • Newing up resources like new mongo.MongoDB(), in the design-time portions of the code, simply uses Mu service discovery to bind to an existing service. This permits capture and use of these so-called capability references to other services.

   • Newing up resources like new mongo.MongoDB() actually provisions them. I would imagine that just running node app.js, where app.js contains the above, would be an incredibly useful and productive way to test out local clusters. Additionally, when the above is deployed into production, should the mu/func's body actually create resources, this should provision them, not discover them.

In addition to all of the above questions, the mode of extensibility must be decided. It would be great if all of this magic can be encapsulated within libraries, rather than needing to, say, host compilers and whatnot. I'm hopeful that the "two ways" approach will enable this.

Although having a bullet-proof DSL for Mufiles is not part of our initial milestones, it is certainly part of our demo scripts. Many tough challenges exist here.

Consider more flexible workspace layout

At the moment, we use a Java- and Go-like approach to mapping module structure to directory structure. Namely, given a stack's, ns1/ns2/.../name, we expect a mirrored directory structure:

<workspace>
|    ns1/
|    |    ns2/
|    |    |    .../
|    |    |    |    name/
|    |    |    |    |    Mufile.yaml

This is a good default. It's super simple to resolve names to locations on disk, without probing or complicated rules. That said, it suffers from the same problems Java packages do (verbosity).

We already envision supporting a namespace section in the optional workspace.yaml file. For this example, we'd set namespace: ns1/ns2/..., and then the workspace could be simplified to:

<workspace>
|    name/
|    |    Mufile.yaml

However, there are still cases that aren't so beautiful. Consider the many number of aws/... stacks. Each one is essentially just a single Mufile.yaml stack. And yet we need subdirectories for each one. A more reasonable approach might be to set namespace: aws, and then do:

<workspace>
|    DynamoDBTable.yaml
|    S3Bucket.yaml
|    ... and so on ...

I'm leaving this as just something to ponder for the future. For now, we'll stick to the simple model.

Implement schema extensibility

In our current sketches, we claim to support fully extensible schemas for stack properties. This allows simple typechecking like strings, bools, and the like. We also want to support arrays, e.g. [ string ], [ bool ], etc., in addition to custom structures like:

type:
    id: number
    name: string

And so on. See https://github.com/marapongo/mu/blob/master/docs/metadata.md for more examples.

There are also open design questions around whether a stack should be able to export schema types. And the relationship with the ongoing API system's design, which presumably also exports schema, is currently unclear, although it would be ideal if they weren't hugely divergent.

Emulate "var" scoping in MuJS

We need to support old-style "var" scoping of variables in MuJS. At the moment, although we track a variables varness, we actually use "let" style scoping for everything.

Precise line/column error reporting

All of the diagnostics infrastructure is in place to offer precise line/column error reporting.

However, due to our simplistic use of stock JSON/YAML parsers, we don't have the hooks necessary to retain this information during unmarshaling. As a result, few AST types implement the diag.Diagable interface, which is the way to feed these to the diagnostics layer. We need to fix this. In the limit, every AST type should have precise information about the range of chars behind it.

lumijs: Emulate ===, !===, >>>, and >>>= operators

At the moment, we map === to ==, !== to !=, >>> to >>, and >>>= to >>=, in the translation into IL. Eventually we presumably need to emulate the ECMAScript semantics of these using runtime helpers.

Finalize the Security design: Users, Roles, and Groups

We have a sketch of what identity (Users, Roles, and Groups) look like in our various design documents, especially the Security design note. We need to sketch that out a bit further and actually implement it in the metadata formats, compiler, and targets.

Serverless events

As we dig deeper into serverless programming patterns, we will want to explore first class abstractions for programmatic events. To start, we can project the raw object types just as they are in the AWS and other cloud provider APIs. But I suspect we will want to go further than this.

For example, in conjunction with #47, this can make serverless really shine. For instance, imagine we want to express a Lambda that processes events coming from an S3 bucket:

let buck = new aws.s3.Bucket(...);
buck.onNewObject((obj) => {
    // deal with the obj event
});

This fictitious example hooks up a lambda, expressed as an ordinary lambda in our favorite language, to an event stream attached to a resource, in this case onNewObject defined on the aws.s3.Bucket stack type, which is presumably of a new primitive type mu.Event.

Expressive diagnostics

After we have done #14, we can go a step further and add "expressive" diagnostics, a la Clang. This helps with usability.

Implement AWS cluster provisioning

Before deploying stacks themselves, we need to be able to create a fresh cluster instance in AWS. This includes setting up the various VPC, subnet, internet gateway, and so on, resources.

Support for sugared semvers

NPM supports convenient shortcuts (like ~ and ^) in its semvers (more here). We should consider supporting these, not the least reason of which being that they are familiar to many developers. These simply desugar into lengthier, but standard, semver ranges.

This could be done as a separate library that we can OSS independently.

mu plan: Implement MuGL plan generation

As soon as we have a MuGL format (#39), and our first MuJS compiler (#40), we need to implement plan generation. This will evaluate the MuPack/MuIL and simultaneously perform diffing against an existing graph to orchestrate the necessary CRUD calls to resource providers (#38).