aws / jsii Goto Github PK

jsii-kernel uses [source-map][1] which depends on a web assembly module mappings.wasm. This is okay until we tried to bundle jsii-runtime into a single resource so it can be embedded into jsii native clients (to reduce external deps).

Since mappings.wasm is being dynamically loaded (using fs.readFile), webpack is unable to include it in the bundle.

It should be possible to interact with CDK classes across multiple threads. However, the jsii runtime is single threaded (uses blocking STDIO to interact with the host). This means that at the moment, if multiple threads attempt to interact with CDK classes, they will likely corrupt the runtime protocol.

At a minimum, we should issue an error if access from multiple threads is attempted

WebAssembly is gaining traction, if we can execute it in all target languages and source languages can compile to it, then it might be an awesome common runtime for jsii modules. Probably still early to decide (copy @buffovich, #83)

@mpiroc does that make sense?

See docs here: https://github.com/Microsoft/TypeScript/wiki/Using-the-Compiler-API

wontfix

To keep a high bar on code quality (at least for constructs we publish)

In .NET, a fully-qualified type name must not collide with a fully-qualified namespace name. The generator attempts to detect these collisions (and rename the type to avoid them). To do this, it needs to know every possible namespace that could cause a conflict.

For root namespaces, this is simple: just inspect the targets property of the assembly itself as well as each of its dependencies (recursively). However, a type can also declare a sub-namespace by including the sub-namespace in its fqn. So we also search types and externals for any sub-namespaces.

The problem arises when a type in my-dependency declares a sub-namespace, but that type is not referenced from my-assembly (and therefore does not appear in externals). The generator has no way to be aware of this sub-namespace, so it can't detect conflicts.

When the generated .NET code for my-assembly is compiled, it takes a reference on my-dependency. So every namespace in my-dependency (even those not referenced from my-assembly) is available to the compiler, and can cause conflicts.

Previously this wasn't a problem, because all external types were present in dependencies. Now that we've switched to externals, this is no longer the case.

We should detect the case where the interface already has the I- prefix, and skip adding an additional I- prefix in this case.

Open question: What if the interface is named something like IAMFoo? I.e. it begins with I<capital letter>, but the I isn't intended as a prefix.

When making a cross-language call that ends up throwing an exception, the stack trace only contains the NodeJS section, and not the host application section. The same applies when a stack trace is internally gathered (for example, to track construct creation locations).

The runtime needs to pass a list of stack frames to the kernel so the exception stack trace can be enhanced to include all relevant information. The kernel then will need to somehow make this external trace a part of the regular trace, so that no special handling is required at stack trace collecting sites.

This could be done by runtime-generating one or more methods, just for the sake of adding stack frames in the Node process. It may also be possible to leverage the fact that almost any string can be a valid method name (so as to only use one stack frame instead of several).

Currently it's not possible to parse a response from the jsii-runtime into a typed structure without your parser knowing ahead of time what type it is expecting. It would be great if the jsii-runtime would include enough information in the response to know what type the response is, then a parser could parse it into a typed structure without relying on out of band information.

This could be as simple as just adding an extra key alongside the ok key, that included that information in it.

In .NET, access modifiers must match exactly. So this is invalid:

class Foo {
    protected abstract void DoSomething();
}

class Bar : Foo {
    public void DoSomething() { }
}

When jsii detects inconsist access modifiers, it should report an error.

See #73

This can be problematic in languages like Java:

class Base {
  Foo: string
}
 
class Derived extends Base {
  Foo: number
}

@mpiroc wrote:

Jsii supports both synchronous and asynchronous overrides. The request to 'complete' an async callback looks like this:

{
    api: "complete",
    cbid: ...,
    error: ...,
    result: ...,
}

But the request to 'complete' a synchronous callback looks like this:

{
    complete: {
        cbid: ...,
        error: ...,
        result: ...,
    }
}

It would simplify backend implementations if synchronous callbacks had the same shape as synchronous callbacks.

https://dev.doctorevidence.com/how-to-write-a-typescript-transform-plugin-fc5308fdd943

Since jsii understands the API of a module, we can determine if there were breaking changes between two versions of the API. This is a very powerful capability and we should introduce it early in the game. This also ties nicely into the desire to review API changes via the normal code review process.

The low-level building block consists of a capability to compare two jsii specs and determine if the changes are minor or major based on the semver specification. The india library implements a similar approach based on these rules (maybe we can use india or extends it somehow to support jsii specs).

Now that we have the ability to compare two jsii specs, we should consider a mode for the compiler to automatically detect breaking changes. This will require checking-in the jsii spec of the module into source control and adding an option to fail the build in case breaking changes are introduced. The workflow (which is not flushed out yet) should also support enforcing version bumps based on the type of changes. See semantic-release - perhaps we can plug into it somehow so this will integarte with our release process.

Currently with the jsii-runtime, whenever the runtime is started it emits a HelloResponse message. This makes sense to ensure that the runtime version is compatible with the client. However, the way it is currently implemented means that the protocol layer (either implicit or explicit) has to specially handle this message.

The typical flow when interacting with the jsii-runtime is essentially a request/response cycle, for every response you get you had to first make a request. However, the hello message is different, it is a response without a correlating request, which means that the protocol layer has to handle a request-less response for just this one single message.

It seems like it'd be sane to instead make it just another command that can be invoked. So instead you'd end up doing something like:

> {"api": "hello"}
< {"hello": "[email protected]"}

We could mimic the command structure even more, and move the {"hello": "..."} into a {"ok": {"hello": "..."}} structure.

This would allow clients to treat every interaction with the jsii-runtime as a request/response cycle, eliminating the need to special case the protocol. Besides the obvious compatibility issues, the biggest issue I see with this is that because this is required to be stable, if we change the framing or format in some way, this command would have to be excluded from that, so you'd end up special casing the parsing of the response for this command instead.

Because of the nature of this command, it would have to be stable for basically forever, but the rest of the protocol can evolve as required.

This would also allow some minor amount of startup optimization for implementations that bundle a jsii-runtime, because the bundling will prevent version drift that this seeks to eliminate.

Thoughts?

At the moment, the jsii toolchain covers the process until a generated code package in all supported programming languages. This leaves the actual build to users, which is unneeded since this is boilerplate that needs to be implemented by all users.

The toolchains required for all languages is expected to be installed on your system (and we will create a docker image that contains all of them).

A few other things to consider:

1. What is the post-build publishing story? Perhaps this is actually part of the "packaging" stage.
2. We need to also make sure that the development environment for contributors doesn't require the entire toolchain because that will become a barrier for contributors. Perhaps the actual multi-language build will only happen during "pack" and contributors can just utilize our build servers to do that for them instead of having to install the entire toolchain.
3. It should be possible to also deprecate a package across all package managers.

tar may not be available on all platforms, and shelling out to it is hence dangerous. The jsii-kernel should instead use a node implementation such as node-tar.

Consider modifying the jsii compiler to inject runtime type checking into the emitted javascript code.
We should consider using ts-runtime by fabiandev (based on his MA thesis).

We can verify everything based on the jsii type info that's bundled within the module:

1. Names of types
2. Names of methods/properties
3. Number of arguments
4. Types of arguments
5. Existence and types of object properties
6. Return types

https://dev.doctorevidence.com/how-to-write-a-typescript-transform-plugin-fc5308fdd943

Since we now encourage using namespaced imports import lambda = require('@aws-cdk/aws-lambda'), it should be fine (at least from JavaScript's perspective) to use Function as the name of the class. If this proves problematic in other languages, jsii generators should mangle the name (similarly to other reserved words).

Also, clean up the class names in @aws-cdk/aws-lambda not to include a Lambda prefix

Ideally we’d like the various language-specific runtime libraries to be self contained and not require node.js installed on the system. This means that we should bundle node.js itself with the libraries for all target platforms, and choose the right one when starting the jsii-runtime.

We can use something like nexe to bundle the runtime and node together, instead of separately.

While methods are fine to return void, they cannot return a Promise<void>, as JSII interprets this as Promise<unknown type>.

Reproduction:

export class PromiseOfVoid {
  async function returnsVoidPromise(): Promise<void> {
    return;
  }
}

Measure and start enforcing test coverage

Please make sure to add a unit test for cover this

See #73

An abstract class with a method of

method(): string | Token

Or whatever. Was accepted by JSII but didn't yield compilable Java code, so should be rejected earlier.

At the moment, an array property is implemented as a List in java. This gives the impression that you could do something like obj.getProperty() and then modify the list. But in reality, this is not a mutable property. The TypeScript behavior is mutable (e.g. obj.Property.push(x). This means that the "correct" approach would be to implement class that derives from List and will affect the underlying javascript array.

Subclasses of Error cannot be exported by a JSII module, since Error is not a JSII class.

Repro:

export class CustomError extends Error {
  constructor(message: string) {
    super(message);
  }
}

We should make sure jsii automatically copies jsdocs if a class has a member that overrides a member in one of the base classes and doesn't have docs.

Assuming an enum type E that has string values, if a method accepts an argument of type E, the JSII specification will incorrectly represent the argument as a string.

In .NET, an override of a get-only property may not contain a setter. So this is invalid:

class Base {
    public abstract string Foo { get; }
}

class Child : Base {
    public override string Foo {
        get => /* ... */ ;
        set => /* ... */ ;
    }
}

When jsii detects inconsistent accessors, it should report an error.

The TypeScript compiler will warn if there is a type mismatch, but we don't get this benefit in runtime. This has a few implications:

1. For javascript (not typescript) users.
2. For users of other dynamic languages.

Ideally, if we could find a way to automatically inject runtime type checks to methods, that would be awesome. Otherwise, we will need more rigour around surface area type checks.

Here's a library that does exactly that: https://github.com/fabiandev/ts-runtime

Various host languages have varying degrees of support for co-existence of multiple versions of the same module within their runtime closure. For example, node/npm simply allows that (since the runtime closure is a directory tree), Java supports it only with certain class loaders, etc.

The questions are:

1. Should the jsii-runtime support it (at the moment, it intentionally blocks this from happening)
2. What happens in environments where this is not supported? Do we have a way to help users resolve these conflicts?

We're still doing something wrong in configuring our package build.

When I run tsc in a directory, it borks with this error. It doesn't when I run jsii, but that's because JSII has its own source discovery mechanism which is distinct from what tsc would do.

This is probably because our package.json:"types" entry points to a .d.ts file, instead of the source .ts file.

If we want to do a TypeScript -> JavaScript+Type Annotations build, we should probably do it out-of-source and rewrite the package.json before publishing.

To allow jsii code to read ARGV from process.argv, during the initial kernel handshake, the host process' ARGV should be passed and the kernel should mutate the sandbox's argv to match.

jsii assemblies have a top-level "readme" field. The java implementation should generate a package-info.java file with this content.

Proposed by @RomainMuller

In .NET, return types must match exactly. So this is invalid:

class A {}
class B : A {}

class Base {
    public virtual A MyProp { get; set; }
}

class Child : Base {
    public override B MyProp { get; set; }
}

When jsii detects inconsistent return types, it should report an error.

Repro: change a few CDK packages to use the same java name and see it blow up.

Description

• Go as a target language (collaboration with JetBrains)

Progress

• Tracking Issue Created
• RFC PR Created aws/aws-cdk-rfcs#206
• Core Team Member Assigned
• Initial Approval / Final Comment Period
• Alpha: https://github.com/aws/jsii/releases/tag/v1.15.0
• Dev Preview: https://github.com/aws/jsii/projects/2
• GA

Per guidance

Definitely an important base type, but remember that it is not natively supported in ES5, so a polyfill will be needed. Still.

Repro: in Getting Started, follow the guide until you reach the "cdk diff" stage.

The output is expected to be:

[~] 🛠 Updating MyQueueE6CA6235 (type: AWS::SQS::Queue)
 └─ [~] .VisibilityTimeout:
    └─ Old value: 300
    └─ New value: 500

But the output is:

[~] 🛠 Updating MyQueueE6CA6235 (type: AWS::SQS::Queue)
 └─ [~] .VisibilityTimeout:

It is legal in TypeScript but won't work in many languages

jsii should provide a predictable style of surface area for many programming languages with multiple styles. It will be beneficial to enforce a convention for naming at the jsii layer, so language-specific generators will be able to rely on that convention to generate idiomatic conventional surface area for their language.

I think jsii should enforce the JavaScript conventions:

• Class names: pascal case
• Properties: camel case
• Methods: camel case
• Arguments: camel case
• Consts: all caps

Brought up by @kiiadi - our beautiful Java "step" builder, which enforces required fields before optional fields will cause backwards compatibility issues when a field is changed from required to optional (which is not supposed to be a breaking change).

Furthermore, the current approach will also cause a breaking change if the order of properties change across revisions, which is also unacceptable.