GithubHelp home page GithubHelp logo

smithy-lang / smithy Goto Github PK

View Code? Open in Web Editor NEW
1.7K 28.0 200.0 128.93 MB

Smithy is a protocol-agnostic interface definition language and set of tools for generating clients, servers, and documentation for any programming language.

Home Page: https://smithy.io

License: Apache License 2.0

Java 81.00% Shell 0.12% Dockerfile 0.02% Smithy 18.84% Batchfile 0.01% Makefile 0.01%
smithy smithy-models

smithy's Introduction

Smithy Smithy

Build Status

Smithy defines and generates clients, services, and documentation for any protocol.

Smithy IDL

Smithy models define a service as a collection of resources, operations, and shapes.

$version: "2"

namespace example.weather

service Weather {
    version: "2006-03-01"
    resources: [City]
    operations: [GetCurrentTime]
}

resource City {
    identifiers: { cityId: CityId }
    read: GetCity
    list: ListCities
    resources: [Forecast]
}

// See the full example at https://smithy.io/2.0/quickstart.html#complete-example

Find out more about modeling a service with Smithy in the Quick Start guide.

Building Smithy models

Important

Before you proceed, make sure you have the Smithy CLI installed.

The Smithy CLI is the easiest way to get started with building Smithy models. First, create a smithy-build.json file:

{
    "version": "1.0",
    "sources": ["model"]
}

Next, create your first model model/main.smithy:

$version: "2"

namespace com.example

service ExampleService {
    version: "2020-05-27"
}

Finally, run smithy build to build the model with the Smithy CLI.

Find out more about building artifacts of your Smithy model in the Building Smithy Models guide. For more examples, see the examples repository

License

This library is licensed under the Apache 2.0 License.

smithy's People

Contributors

82marbag avatar aajtodd avatar adamthom-amzn avatar alextwoods avatar andrewfossaws avatar crisidev avatar david-perez avatar davidogunsaws avatar dependabot[bot] avatar gosar avatar haydenbaker avatar hpmellema avatar ianbotsf avatar jasdel avatar jdisanti avatar jordonphillips avatar jvschneid avatar kggilmer avatar kstich avatar miguel-vila avatar milesziemer avatar mtdowling avatar rchache avatar rcoh avatar skmcgrail avatar skotambkar avatar srchase avatar sugmanue avatar sullis avatar syall avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

smithy's Issues

Improve JSON AST model format

The JSON model format was originally intended to be authored through YAML or Ion files, and as such, we made tradeoffs to make it less verbose. For example, namespaces are just extra top-level keys in the model, traits are extra keys in shapes, etc. Now that we have an IDL format and recommend people author models with that, we could look at improving the JSON format to be easier to parse, easier to load into a semantic model, and easier to grep.

Version number

I recommend that we create a version 0.5.0.

Backward compatibility

We will continue to support loading 0.4.0 JSON AST models and emit deprecation warnings when they are encountered. We will remove support for 0.4.0 when 1.0.0 is released.

Explicit namespaces

Namespaces are currently defined as top-level additional properties.

{
    "smithy": "0.4.0",
    "smithy.example": {
        "shapes": {
            "MyString": {
                "type": "string",
                "documentation": "My documentation string"
            }
        }
    }
}

This is problematic because:

  1. It's generally harder to parse this kind of schema or interact with it in tools like jq.
  2. We might need to add other properties to Smithy models, and relying on additional properties could technically result in a breaking change if a namespace uses the same name as an added property.

I recommend that we add a top-level "shapes" property that contains shape definitions.

{
    "version": "0.5.0",
    "shapes": {
        "smithy.example#String": {
            "type": "string"
        }
    }
}

Absolute shape IDs

The new JSON format will support only absolute shape IDs both when defining and referencing shapes.

{
    "version": "0.5.0",
    "shapes": {
        "smithy.example#String": {
            "type": "string",
            "traits": {
                "smithy.api#documentation": "docs"
            }
        }
    }
}

Traits are stored in the traits property

In the 0.4.0 format, traits are applied to shapes using additional properties on shapes. This makes loading shapes harder than it needs to be since you need to keep a set of know shape properties and skip over key-value pairs for known properties to detect traits. Using additional properties also makes it harder to add new properties to shapes in the future since trait names could potentially conflict with these properties (for example, we're considering adding an "isa" property to structures).

As seen in the above example, traits in 0.5.0 are defined on shapes inside of the "traits" property of a shape. This property is a map of absolute shape IDs to the trait's value. Yes, even traits use absolute shape IDs.

Applying traits is done with a type of "apply"

In 0.4.0, traits applied externally to shapes using the "traits" property of a namespace:

{
    "smithy": "0.4.0",
    "smithy.example": {
        "traits": {
            "MyString": {
                "documentation": "This is my string!"
            }
        }
    }
}

However, this makes working with single-file models using tools like grep, jq, or jmespath harder. For example, imagine if you needed to find all shapes with a specific trait. You'd need to search through both the "shapes" property and "traits" property of every namespace.

There's also an issue with how a file stores models. A single JSON model should not both define a shape and externally apply traits to it. For example, this is a sub-optimal model:

{
    "smithy": "0.4.0",
    "smithy.example": {
        "shapes": {
            "type": "string"
        },
        "traits": {
            "MyString": {
                "documentation": "This is my string!"
            }
        }
    }
}

I propose that we reuse the "shapes" map and add a new "type" called "apply" that is used to apply traits to shapes.

{
    "version": "0.5.0",
    "shapes": {
        "smithy.example#MyString": {
            "type": "apply",
            "traits": {
                "smithy.api#documentation": "This is my string!"
            }
        }
    }
}

This now makes it much simpler to find all shapes with a given trait since every trait is defined in a uniform location in the model. It also makes it impossible to both define a shape and apply traits to a shape externally in the same file.

Normalized relationships

Relationships between shapes will use a normalized object format of a property named "target" that maps to an absolute shape ID. While things like service operations, operation input, operation output, and operation errors are not actually member references that allow traits, using an object form that matches the form of members makes it easier to detect shape references in tools like grep, jq, or jmespath.

UPDATE: Rethinking this, I realized that using $ for some shape relationships but not all would actually introduce an inconsistency in the format. For example, trait values with an idRef trait would not use this syntax, nor would applying traits to shapes since a trait would be the key of a map. A better approach for simpler tools to try to reason about references between shapes would be to look for strings that match the shape ID ABNF.

Example

The following example is a bit longer form that defines several relationships using the normalized format.

{
    "version": "0.5.0",
    "metadata": {
        "xy": "z"
    },
    "shapes": {
        "com.foo#Service": {
            "type": "service",
            "operations": [
                {"target": "com.foo#Operation"}
            ]
        },
        "com.foo#Operation": {
            "type": "operation"
        },
        "com.foo#Baz": {
            "type": "structure",
            "members": {
                "foo": {
                    "target": "smithy.api#String",
                    "traits": {
                        "smithy.api#documentation": "Hello"
                    }
                }
            },
            "traits": {
                "smithy.api#documentation": "Hello"
            }
        },
        "com.foo#Baz$foo": {
            "type": "apply",
            "traits": {
                "smithy.api#sensitive": true
            }
        }
    }
}

Add event streams to OpenAPI conversion

We should add support for converting Smithy event streams to OpenAPI. This would likely be a string with the "binary" format and use a media-type of application/vnd.amazon.eventstream (depending on the protocol of course).

Because this wouldn't work with Lambda or API Gateway, we should add the ability to enable or disable this functionality in smithy-build.

Make @collectionOperation trait required for collection operations

It's too easy right now to accidentally create a collection operation. It happens by omitting a resource identifier binding, misspelling a member name, etc. I propose we make it mandatory to use the @collectionOperation trait on operations bound to resources as a collection operation. We should also remove the @instanceOperation trait if this is added since it's logically implied.

Add OAuth 2.0 auth scheme, flows, and scopes

We should add built-in support for OAuth 2.0, including traits used to configure OAuth and specify scopes per/operation. These traits and scopes would also be used in Amazon Cognito auth and when using OpenID Connect.

Tracking: SDK-23710

Add enum support to byte, short, int, and long

The enum trait today can only be attached to a string shape. Updating this trait to also support byte, short, int, and long would make it support a broader range of uses cases that involve a finite set of numeric values.

Add ability to import shapes and traits as aliases

Smithy supports fully qualified shape and trait names (e.g., smithy.example#Foo). It would be nice if the IDL supported the ability to locally alias shapes and traits to shorter symbols in a file. These aliases would not persist between loading and serializing models and would not be supported in the JSON format. (note that these aliases would not work with trait values, including idRef trait values, since this would couple trait loading to a specific model format).

The syntax would resemble:

use shape smithy.example#Foo
use trait smithy.example#hello

Shape definition names and trait definition names defined in a file must not conflict with any imported aliases (for example, you couldn't import Baz and define a shape named Baz in the same file).

Supported references

  • resolving the name of a trait applied to a shape
  • resolving shapes targeted by members
  • operations and resources properties of service and resource shapes
  • create/read/update/delete/list properties of resource shapes
  • operation input/output/errors
  • trait definition "shape" properties
  • apply statement "shape" and trait

What about idRef?

Resolving idRef values inside of traits is far trickier than any of the above. For example, the references trait can target a service or resource shape, and these shapes can potentially be imported into the namespace that the trait was defined in. This isn't too difficult to implement for traits that have a concrete class implementation resolved at runtime (though it would need a big breaking change to the TraitFactory and TraitService interfaces), but would be very difficult to implement for DynamicTrait instances.

DynamicTraits are traits that don't have a known concrete class but have a known trait definition (unless you allow unknown traits). Dynamic traits that contain values for shapes that are marked with the idRef trait is where the complexity comes in.

Resolve the IDs at load time

Resolving dynamic traits at load time would be pretty difficult. DynamicTraits would need to be created during the model loading process then create a fully built (but not validated) Model. Then we'd iterate over all DynamicTraits and see if any idRef traits are applied in the closure of any shapes referenced by DynamicTraits. If so, we'd need to recreate each dynamic trait, rebuild each shape they're attached to, and then rebuild the model. The benefit to resolving shape IDs at load time is that it removes any kind of runtime complexity in dealing with trait IDs, particularly when validating the trait values.

Resolve the IDs at validation/runtime

Resolving dynamic traits at validation/runtime would require that the aliased shapes that existed at the time of creating the DynamicTrait are persisted on the trait. We could then pass in the aliases into the NodeValidationVisitor, which would then be utilized by the IdRefPlugin. The benefit of this approach is that it significantly reduces the complexity of implementation, but pushes this complexity out to consumers of the model.

Don't resolve them at all

The simplest option is to not idRef trait values at all against imports. However, this could result in confusing behavior. Consider the following example. Assume that the references trait is a DynamicTrait and not one of the built-in traits.

namespace smithy.example

use shape smithy.weather#Weather

@references(forecast: {resource: Forecast, service: Weather})
structure ForecastInformation {
  @required
  forecastId: ForecastId,
}

In this example, the Weather shape was imported from smithy.weather. The references trait contains a reference to Weather, but because idRefs don't know about aliases, it assumes that the ID is relative and becomes smithy.example#Weather.

Resolve them in the parser with special syntax

A final option for resolving idRef values it to introduce a special syntax for indicating that a specific value inside of a node is a relative shape ID that should be resolved with aliases. For example, the lexer could consider any unquoted string value that starts with "#" to be a relative shape ID. The parser would then resolve these references when loading trait values.

The following example would correctly treat the reference to #Weather as smithy.weather#Weather, and #Forecast would be resolved to smithy.example#Forecast:

namespace smithy.example

use shape smithy.weather#Weather

@references(forecast: {resource: #Forecast, service: #Weather})
structure ForecastInformation {
  @required
  forecastId: ForecastId,
}

This is the simplest approach that works for idRef and doesn't require any kind of major refactor.

The weirdest thing about this approach is that it requires a special syntax for shape IDs when defining traits but that syntax isn't required for things like members and operation input/output. For example, we'd need to support the following for consistency even though the bar and baz members both target smithy.other#Bar:

namespace smithy.example

use shape smithy.other#Bar

structure Foo {
    bar: Bar,
    baz: #Bar,
}

Resolve them in the parser automatically (selected option)

Instead of using the special # syntax, we could just make unquoted node value strings in traits automatically convert to the fully qualified shape name when a trait value matches the name of an imported shapes. This could be surprising, but the recourse is to quote the string.

In the following example, the value "Bar" in the @foo trait would be converted to smithy.other#Bar because it is unquoted and matches the name Bar.

namespace smithy.example

use shape smithy.other#Bar

trait foo {
    shape: FooTrait,
}

@idRef
string FooTrait

@foo(Bar)  // resolves to "smithy.other#Bar"
string MyString

@foo("Bar")  // Resolves to "Bar", which later, during validation, assumes smithy.example#Bar
string AnotherString

The drawback of this approach is that the replacement of Bar to smithy.other#Bar is indifferent to whether or not the underlying shape has the idRef trait. This means that we'd need to recommend that strings for trait values should be quoted unless you explicitly want to use a shape alias. Also note that this would not support true, false, or null as those are keywords for node values (this would be an extreme edge case that likely doesn't matter at all).

Can we remove the constraint that sets can only target simple types?

Today, set shapes can only target simple types. I'd like to explore allowing sets to target the same shapes as list, but we'll need to understand if that is something that can't be reliably achieved across all supported languages. For example Ruby, Java, C#, and Python allow for custom hashcodes and equality, but PHP and JavaScript do not.

Smithy does not flag self-referencing list/set as invalid

list SomeList {
    member: SomeList,
}

Some shape definitions like above passes Smithy validation without error/warning but in reality they are apparently not valid definition.

Please consider flagging such error in the validation, thanks!

Example for Codegen Workflow

Hi Michael,
the project looks very promising! We have been looking for an alternative to a Swagger-based cross-language codegen and API specification solution to replace our current version.

Do you have plans for a documentation of the codegen parts of this repo or is this out of scope for this project?
Are you using this already, e.g. in the scope of the aws-sdk or are these tools?

Kind regards

Add validator(s) to enforce protocol and auth requirements

We need one or more validators that enforce requirements of protocol or auth schemes. For example, if an auth scheme requires a specific trait on a service, then the validator should be able to capture this and emit when the service does not have the trait. There should be an option to enforce the inverse -- that is, if a trait is present, then it requires that a specific auth scheme or protocol is defined on a service.

Is it possible to share smithy concepts as re-usable modules/packages?

I've been looking at the documentation for Smithy and really like the namespace feature but I don't see any documentation/examples explaining how to structure a project. Does Smithy scan a directory for all smithy files and merge them into a "classpath" like concept? Or must everything be in a single file?

How would I go about packaging up smithy definitions into re-usable modules? E.g. would it be possible to publish shapes/operations/resources/traits for others to include in their API? Are services, resources and operations considered part of a namespace, or just shapes and traits?

Apologies if I missed some obvious documentation.

Add ability to append to map trait values

The apply statement can open up and append to traits that use list values, but can't currently extend map values. Allowing the apply statement to append to map trait values opens up new fan-out use cases where models can be merged together, even conditionally inside of Smithy build, to do things like add auth schemes that are only relevant to specific projections.

Update docs to use 4 spaces for the IDL and JSON

I've started to write newer documentation using 4 spaces for both the IDL and JSON models, and I think it greatly improves the readability of the model. Using two spaces isn't enough of a visual difference for me to easily scan a model (maybe I just have bad eyesight? Ok, I know I do ๐Ÿ˜„).

Optimize ValidationEvent for readability

ValidationEvents formatted as a string are hard to read. Is it possible to make them more succinct or easier to read?

Here's an example of the error format:

[ERROR] (Target) foo.bar#Foo$foo /tmp/foo.smithy [4, 6]: member shape targets an unresolved shape `foo.bar#Missing`

While the above is some we can easily parse into distinct parts, it places some unnecessary information at the front of the message before we can even understand the error. I propose moving the error ID and source location to the end of the message. Something like:

[ERROR] foo.bar#Foo$foo: member shape targets an unresolved shape `foo.bar#Missing` | Target /tmp/foo.smithy:4:6

This is still quite easily parsed (not using a formal grammar, but something that's deterministic):

First, find the last occurrence of "|" and split into two parts: LHS and RHS:

LHS:

  1. [ followed by the severity followed by ]
  2. A shape ID or - followed by :, where - means that there was no shape targeted by the violation.
  3. All remaining text

RHS:

  1. Split based on " ".
  2. The first value is the name of the violation
  3. The second value is the source location filename, followed by ":", the line number, ":" the column.

Here's what Checkstyle looks like:

[INFO] src/software/amazon/smithy/aws/apigateway/internal/openapi/CloudTrailConfiguration.java:67:17: Avoid inline conditionals. [AvoidInlineConditionals]

I like that they place the violation name at the end of the string. Because they deal with filenames and line numbers, it's similar to how in Smithy we would place the shape ID at the beginning of the string.

Note that making this change requires quite a few test cases to be updated.

Smithy artifacts not in maven central

As far as I can tell, it looks like none of the software.amazon.smithy artifacts have been uploaded to maven central. It also looks like the gradle plugin hasn't been uploaded to the gradle plugin repository. This means that the snippet to use the plugin in the README doesn't work as well because gradle fails to find the plugin.

Getting Started - Building and Running CLI with Intellij

This project is very interesting! I'm trying to get Intellij setup to build and run the smithy-cli (I'm assuming smithy-cli is the place to start playing around).

My Grade settings stuff looks like this:

image

My Gradle window looks like this.

image

My Java compiler settings look like this (1.8):

image

When I try to run :smithy-cli:run task I get this:

> Task :smithy-cli:run FAILED
Error: VM option 'SharedArchiveFile' is diagnostic and must be enabled via -XX:+UnlockDiagnosticVMOptions.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Error: VM option 'SharedArchiveFile' is diagnostic and must be enabled via -XX:+UnlockDiagnosticVMOptions.

Error: Could not create the Java Virtual Machine.

Error: A fatal exception has occurred. Program will exit.

Execution failed for task ':smithy-cli:run'.
> Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_211.jdk/Contents/Home/bin/java'' finished with non-zero exit value 1

Am I doing something wrong? Is it my ancient (Amazon installed) JDK version? I haven't used Gradle before, sorry (mostly done C++, Go, Node and stuff like that).

Thanks

[0.10] Emit warnings for operations and resources not bound to any service

When building models that span multiple files, it's easy to forget to bind an operation or resource to a service. We should add a built-in validator that emits warning events when operations or resources are found that are not bound within the closure of a service. This is similar to the UnreferencedShapes validator, but it is run automatically and doesn't check anything other than operations or resources.

warning: Deprecated Gradle features were used in this build

Followed steps from https://github.com/awslabs/smithy-typescript#steps-to-build

Output:

$ ./gradlew publishToMavenLocal
Downloading https://services.gradle.org/distributions/gradle-5.4.1-bin.zip
...................................................................................

Welcome to Gradle 5.4.1!

Here are the highlights of this release:
 - Run builds with JDK12
 - New API for Incremental Tasks
 - Updates to native projects, including Swift 5 support

For more details see https://docs.gradle.org/5.4.1/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)

> Task :smithy-model:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :smithy-jsonschema:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :smithy-build:compileJava
Note: /local/home/trivikr/workspace/smithy/smithy-build/src/main/java/software/amazon/smithy/build/PluginContext.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :smithy-openapi:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :smithy-codegen-freemarker:compileJava
Note: /local/home/trivikr/workspace/smithy/smithy-codegen-freemarker/src/main/java/software/amazon/smithy/codegen/freemarker/FreeMarkerEngine.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :smithy-protocol-test-traits:compileJava
Note: /local/home/trivikr/workspace/smithy/smithy-protocol-test-traits/src/main/java/software/amazon/smithy/protocoltests/traits/HttpMessageTestCase.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :smithy-model:javadoc
/local/home/trivikr/workspace/smithy/smithy-model/src/main/java/software/amazon/smithy/model/Model.java:215: warning: no @param for <T>
    public <T extends Shape> T expectShape(ShapeId id, Class<T> type) {
                               ^
1 warning

Deprecated Gradle features were used in this build, making it incompatible with Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/5.4.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 24s
107 actionable tasks: 107 executed

Add trait code generation

The code used to represent typed values for traits are ripe for code generation-- they're mostly the same thing and formulaic depending on the type of trait.

I pushed up a proof of concept branch to kick start the idea. It needs work and has lots of todos. I probably wont have the cycles to finish this for the next few weeks, so anyone is free to take it over.

https://github.com/awslabs/smithy/tree/smithy-codegen-traits/codegen/smithy-codegen-traits/src/main/java/software/amazon/smithy/codegen/traits

Notes and considerations:

  • While it would be nice to generate the traits in the smithy-model package, it might not be worth it at this point since it would require pulling a ton of code out into a lower-level package that doesn't depend on code generation so that the smithy-model package could.
  • It would be nice if this could be an exercise to prove out the design of CodeWriter, but if that proves insufficient, feel free to pull in a dependency on JavaPoet.
  • Traits that target structure shapes should inline the members of the targeted structure into the trait itself rather than require a structure value to be passed into the code generated trait.

Tracking: SDK-23707

Rehaul strings in the IDL

  • Add support for all JSON escape sequences in all Smithy strings.
  • Add "text blocks" with automatic removal of incidental whitespace, supporting both """ and '''. Pretty much this: https://openjdk.java.net/jeps/355
  • Remove string concatenation where we would handle two strings next to each other: "foo" "bar" -> "foobar"

Deprecate then remove ShapeIndex

ShapeIndex is an abstraction that basically is a map of ShapeId to Shape. A Model contains a ShapeIndex, metadata, and a version. A ShapeIndex as a separate abstraction from Model makes code harder to write and adds no value in its separation. I propose we release a backward compatible change that first adds the ShapeIndex API into Model and deprecates any ShapeIndex based API. Then, in the next major version (i.e., 0.10.0), we will remove ShapeIndex and the deprecated APIs. This will take probably several days of super lame find, replace, and edit changes, but I think it's a win for API clarity.

  • Deprecate ShapeIndex in 0.9.x
  • Remove ShapeIndex in 0.10.x

Drop "generate" command from CLI

The generate command in the CLI should be replaced with just using the build command and passing in arguments to filter it.

For example, you should get the same functionality as the generate command by doing:

echo "{...}" | smithy build -c - --projection X --plugin Y --output /path/to/z

Which would then load the contents of a smithy-build.json from stdin.

Clean up IDL use and namespaces for 0.10.0

The IDL currently allows multiple namespaces to be defined in a single file and for use statements to appear before a namespace is defined. This is strange and hard to understand:

  • Should changing a namespace cause previously defined use statements to go away? The answer right now is no, but that requires looking at the spec.
  • Should use statements be allowed before entering a namespace? The answer right now is yes, but that probably requires looking at the spec.

I propose we clean this up and simplify the IDL in 0.10.0:

  1. Only a single namespace can be defined per IDL file.
  2. Use statements can only appear after a namespace has been defined.
  3. Use statements cannot import shapes from the same namespace into the current namespace.
  4. Metadata statements must come before any namespace statements.
  5. Only a single version statement is allowed per file.

These changes will make it easier to parse and understand the Smithy IDL, particularly as we look to build out IDE integrations in 2020.

Update Smithy documentation strings to support CommonMark

Smithy's documentation strings currently expect HTML formatting. Many modern programming languages (like Rust, Idris, and Crystal) and specification formats (like OpenAPI, RAML) support Markdown as the syntax for documenting constructs. Most existing Smithy models (and formats that convert to Smithy internally at Amazon) utilize some form of HTML or DocBook to document shapes.

Given that CommonMark explicitly calls out HTML blocks as a way to use raw HTML, this change would most likely be backward compatible with any existing documentation strings. The biggest difference would be that tooling consuming Smithy models would need to convert Markdown strings into HTML to present them in documentation.

The commonmark-java library from Atlassian looks quite nice and has no dependencies.

Timestamp validation is technically wrong (Java 12 test failures)

There was a bug in DateTimeFormatter.ISO_INSTANT that was fixed in Java 12. Here is the link to that bug:
https://bugs.openjdk.java.net/browse/JDK-8166138

This change exposes certain tests in

smithy-model/src/test/java/software/amazon/smithy/model/validation/NodeValidationVisitorTest.java

which rely on the incorrect, pre Java 12 behavior of DateTimeFormatter.ISO_INSTANT
For example, it is asserted that the following string should fail timestamp validation:

2000-01-12T22:11:12+01:02

In fact, this is a valid RFC 3389 time representation.
In Java versions prior to 12, DateTimeFormatter.ISO_INSTANT incorrectly throws for this parse, and the test which relies on this incorrect behavior passes.
In Java version 12, the Java SDK bug was fixed and the test fails because the string is a valid timestamp and Java no longer throws on the parse.

Bottom line: there are some tests that fail in Java 12.

Make aws.api#unsignedPayload trait an annotation trait

The aws.api#unsignedPayload trait currently takes a set of signing names to enable unsigned payloads on. However, I don't think we will ever need this level of granularity and it makes it more difficult to implement codegen.

Remove synthetic trait in favor of service errors

The synthetic trait has been a bit confusing to developers. It automatically registers an error with every operation. However, there's no way to selectively specify which errors a service could actually encounter / filter out specific synthetic errors.

I propose that we remove the synthetic trait and instead add an errors property to the service shape just like the errors property of an operation. This will associate the error with every operation bound to the service and is much more explicit.

  • Remove synthetic trait
  • Add errors to service

Add "///" to define documentation for shapes and traits in IDL

It would be nice to support using /// to define documentation traits for shapes and trait definitions. This would require that the values for /// are buffered until a shape or trait definition are encountered, and then they'd be applied as either a @documentation trait to a shape or as a documentation property to a trait definition.

For example:

namespace smithy.example

/// Defines a shape as a "foo".
/// More text that's added to the previous line...
trait foo {
    selector: "*",
    shape: Baz,
}

/// The value of a foo trait.
string Baz

Is equivalent to:

{
  "smithy": "1.0",
  "smithy.example": {
    "traitDefs": {
      "foo": {"selector": "*", "shape": "Baz", "documentation": "Defines a shape as a "foo".\nMore text that's added to the previous line..."
    },
    "shapes": {
      "Baz": {"type": "String", "documentation": "The value of a foo trait."}
    }
  }
}

Document "dataplane" tag

Various abstractions used internally and externally need to know if an operation is considered part of the control plane or data plane.

  • The control plane is responsible for exposing the APIs and interfaces used to define, deploy, and lifecycle resources.
  • The data plane is responsible for providing the APIs and interfaces for sending messages to resources.

We use the "dataplane" tag to mark an operation as part of the data plane. Other operations are not explicitly considered data plane and are assumed to be part of the control plane.

Don't validate private shapes with most linters unless you opt-in

When building the smithy-aws-traits package with validation from the smithy-linters package, you get the following warnings:

[WARNING] aws.apigateway#Response$responseParameters: The `Response` structure shape stutters its name in the member `responseParameters`; structure member names should not be prefixed with the structure name. | StutteredShapeName
[WARNING] aws.apigateway#Response$responseTemplates: The `Response` structure shape stutters its name in the member `responseTemplates`; structure member names should not be prefixed with the structure name. | StutteredShapeName

I think it would be useful to update lots of the existing linters to not validate shapes marked as private unless you explicitly pass something like "validatePrivateShapes".

Show test failure output on Travis

When tests fail on Travis, it doesn't actually show the output of the failure. It expects you to open the HTML output, but that's not possible since it's running remotely on Travis. We need to update the test runner through Gradle to output test failure information to the console.

@retryable should specify how to retry

Excessive retries can lead to problems, especially when the operation is failing because it is already overloaded. I'm looking at the retryable example from the spec:

@error("server")
@retryable
@httpError(503)
structure ServiceUnavailableError {}

Without a specific strategy this may lead to retry storms. One way to get around this would be to specify strategies for retrying, for example:

@error("server")
@retryable(strategy: "TokenBucket") // Retries once for every 1000 successful requests
@httpError(503)
structure ServiceUnavailableError {}

@error("server")
@retryable(strategy: "ExponentialBackoff", times: 3)
structure FooError {}

@error("server")
@retryable(strategy: "OnceImmediately")
structure BarError {}

Of course best practices should be the default if nothing is specified, but in some cases a different design is warranted, and a one-size-fits-all approach doesn't make sense

Tracking: SDK-23706

Symlinks are added to test resources during test and breaks builds

Symlinks appear to be added while testing that the ModelAssembler can follow symlinks. They are currently added directly to the test directory rather than a temp directory. Alternatively, after a test is finished or crashed, the symlinks should be removed.

Could not list contents of '/[.snip.]/smithy/smithy-model/src/test/resources/symlink-nested'. Couldn't follow symbolic link.

Related to 3fe63b0

SmithyBuildTest is non-deterministic

Random test cases in SmithyBuildTest seem to occasionally fail. For example, from a Gradle build:

software.amazon.smithy.model.build.SmithyBuildTest > buildCanOverrideConfigOutputDirectory FAILED
    java.lang.AssertionError:
    Expected: iterable with items [</custom/foo/source/model/model.json>, </custom/foo/source/build-info/smithy-build-info.json>, </custom/foo/a/model/model.json>, </custom/foo/a/build-info/smithy-build-info.json>, </custom/foo/b/model/model.json>, </custom/foo/b/build-info/smithy-build-info.json>] in any order
         but: no item matches: </custom/foo/source/model/model.json>, </custom/foo/source/build-info/smithy-build-info.json> in [</custom/foo/a/model/model.json>, </custom/foo/a/build-info/smithy-build-info.json>, </custom/foo/b/model/model.json>, </custom/foo/b/build-info/smithy-build-info.json>]
        at [email protected]/org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at [email protected]/org.junit.Assert.assertThat(Assert.java:956)
        at [email protected]/org.junit.Assert.assertThat(Assert.java:923)
        at software.amazon.smithy.model.build/software.amazon.smithy.model.build.SmithyBuildTest.buildCanOverrideConfigOutputDirectory(SmithyBuildTest.java:277)

And here's a Maven build:

[INFO] Running software.amazon.smithy.model.build.SmithyBuildTest
[ERROR] Tests run: 13, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.53 s <<< FAILURE! - in software.amazon.smithy.model.build.SmithyBuildTest
[ERROR] appliesPlugins(software.amazon.smithy.model.build.SmithyBuildTest)  Time elapsed: 0.068 s  <<< ERROR!
java.util.NoSuchElementException: No value present
	at software.amazon.smithy.model.build/software.amazon.smithy.model.build.SmithyBuildTest.appliesPlugins(SmithyBuildTest.java:236)

Add validator to mark protocols or auth as deprecated

We need a validator that can be used to mark protocols or authentication schemes as deprecated. This can be a single validator or two different validators and the settings value should accept a list of protocol names / auth scheme names.

How to generate clients for AWS services

I am not sure whether my understanding of smithy is correct.
Can it be used to generate clients for e.g. S3 / SQS for a programming language like D?
Where are the s3 / sqs smithy model files located?

How are the aws-crt-* libraries related to smithy? Are they the base for generating the clients / servers?

Improve trait coercion for forward compatibility

By making trait coercion a bit more relaxed, we can make it easier to forward compatibly evolve traits over time. I propose we support the following use case of changing a trait from a boolean trait to a trait that has a structure, union, or map value. We could support this by accepting both null and true for structure, union, and map traits (null because it allows for empty traits in the IDL that have no value).

This would allow, for example, us to add properties to something like the retryable triat to indicate that it is also throttling.

Some sort of trait abstraction capabilities?

I'm finding myself writing nearly the same @paginated all over my smithy code because I want the same token/maxresults variables and consistency across my API. The only thing that meaningfully varies is the items field.

I was wondering if it was currently (or could become) possible to have some sort of @copumpkinPaginated(items: Foo) where I define @copumpkinPaginated in one place and then reuse it everywhere else, and it expands to @paginated with the right arguments.

More generally it seems like some sort of basic macro system would be nice. I've considered layering even the C preprocessor over smithy to avoid writing all the XRequest XResponse structures out by hand.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.