GithubHelp home page GithubHelp logo

aws / smithy-go Goto Github PK

View Code? Open in Web Editor NEW
158.0 14.0 50.0 2.01 MB

Smithy code generators for Go (in development)

License: Apache License 2.0

Go 33.67% Java 65.47% Makefile 0.18% Shell 0.04% Smithy 0.64%
smithy golang

smithy-go's Introduction

Smithy Go

Go Build StatusCodegen Build Status

Smithy code generators for Go.

WARNING: All interfaces are subject to change.

Can I use this?

In order to generate a usable smithy client you must provide a protocol definition, such as AWS restJson1, in order to generate transport mechanisms and serialization/deserialization code ("serde") accordingly.

The code generator does not currently support any protocols out of the box, therefore the useability of this project on its own is currently limited. Support for all AWS protocols exists in aws-sdk-go-v2. We are tracking the movement of those out of the SDK into smithy-go in #458, but there's currently no timeline for doing so.

License

This project is licensed under the Apache-2.0 License.

smithy-go's People

Contributors

aajtodd avatar aknuds1 avatar alrs avatar amazon-auto avatar anthonyfok avatar eddy-aws avatar eduardomourar avatar haines avatar isaiahvita avatar jasdel avatar jordonphillips avatar lucix-aws avatar madrigal avatar milesziemer avatar ranvaknin avatar rcoh avatar shogo82148 avatar skmcgrail avatar skotambkar avatar syall avatar wty-bryant avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

smithy-go's Issues

Int overflow in time_test.go on 32 bits arches

With Go 1.15.6, when running the tests on i686 and armv7hl:

github.com/aws/smithy-go/time
# github.com/aws/smithy-go/time [github.com/aws/smithy-go/time.test]
./time_test.go:9:50: constant 50520000000 overflows int
./time_test.go:45:49: constant 21123399936 overflows int
FAIL	github.com/aws/smithy-go/time [build failed]

`xml.WrapNodeDecoder.Token` incorrectly terminates on nested tags

Consider:

<Response>
   <Response>abc</Response>
   <A/>
</Response>

WrapNodeDecoder.Token when passed the outer Response as the start element should return two Start elements, Response and A. But instead, </Response> confuses the decoder and <A/> is never returned.

func Test_NestedElements(t *testing.T) {
        responseBody := bytes.NewReader([]byte(`<Response><Response>abc</Response><A/></Response>`))

        xmlDecoder := xml.NewDecoder(responseBody)
        // Fetches <Struct> tag as start element.
        st, err := FetchRootElement(xmlDecoder)
        if err != nil {
                t.Fatalf("Expected no error, got %v", err)
        }

        // nodeDecoder will track <Response> tag as root node of the document
        nodeDecoder := WrapNodeDecoder(xmlDecoder, st)
        for {
                token, done, err := nodeDecoder.Token()
                fmt.Println(token, done, err)
                if done {
                        return
                }
        }
}

Consider preallocating middleware order slices/maps

The middleware's orderedIDs and relativeOrder types use maps and lists to keep track of the ordering of middleware within a stack. Should consider if these orders should be pre-allocated to reduce further allocations, since multiple middleware in each step will almost always be added.

In addition the Insert, Before case should handle the case where the capacity of the list is greater than its length to prevent allocating multiple new arrays.

Why is v0.1.0 the latest available version of the plugin?

AFAICT, v0.1.0 is the latest version of the plugin that has been published, even though the latest release on GitHub is v1.5.0. At least Gradle can only find v0.1.0. Can anyone explain how come newer versions aren't published?

API client operations should allow nil input value

The API client operations should allow the user to pass in nil for the input parameters. This removes the need for the users to experience unexpected nil-dereferences in the SDK, and makes it more straight forward when making API calls with only optional parameters.

func (c *Client) BatchWriteItem(ctx context.Context, params *BatchWriteItemInput, optFns ...func(*Options)) (*BatchWriteItemOutput, error) {

	if params == nil {
		params = &BatchWriteItemInput{}
	}

	stack := middleware.NewStack("BatchWriteItem", smithyhttp.NewStackRequest)

It would also be nice to generate custom doc letting user know this is an option for operations with only optional parameters.

Minor Code Improvements

Good find. I think this logic must have been duplicated elsewhere.

Also something phjordon@ suggested to me few weeks back - that I liked.

String v = String.format("payload := strings.NewReader(%s)", payloadShape.hasTrait(EnumTrait.class)? "string($L)": "*$L");
writer.write(v);

Originally posted by @skotambkar in #296 (comment)

Include `json` tag in structs if field is annotated with @jsonName

As title says, in its current form json tags are not included in the generated Go code.

We use jsonName extensively throughout our Smithy models, but the Go code is unusable without the json struct tags.

If you can point me towards where it needs to be added, I can look at submitting a pull request.

Many thanks.

Lists members should be generated as values by default

Member's of List shapes should not be generated as pointer values by default. The members of lists should be generated as values.

This changes how the SDK generator and serializers handle list/set shapes. During deserialization the SDK should substitute the type's zero value for null values, and during serialization send the value of the list member.

Go inputs/outputs can't be serialized/deserialized into valid smithy

The StructureGenerator creates input and output types for smithy actions.
Unlike in other languages, the fields in the generated types are capitalized. This is done because they need to be exported and go uses capitalization for this.

A consequence of this is that the types can't be simply serialized with go's default serialization library as the capitalized field names won't necessarily match what smithy defines.

These input/output types are not strictly required to be serializable by default, but it would make working with the go types generated by smithy-go a lot easier. This could be quite easily supported by simply adding struct tags with names matching those defined in smithy to all the capitalized fields generated by the structure generator.

It seems like the smithy-go team explicitly doesn't want these types to be used in serialization/deserialization based on the noSmithyDocumentSerde that is generated. What's the reasoning for this?

Just trying to get a better understanding of the intent to better shape our use of smithy-go.

Asking because for our use-case having the structures generated with struct-tags matching the smithy names would be very convenient and useful but it seems like this is something smithy-go explicitly wants to exclude.

Examples of generated API client usage?

Do there exist any examples of generated API client code usage? I've been able to generate an API client for some Grafana endpoints, but can't yet figure out how to e.g. configure the API server address.

API client's should remove APIOptionsFunc type alias

The type alias generated into API Client package, APIOptionsFunc should be removed because it makes it harder for utility functions defined outside of the API client's package to be compatible with the APIOptionsFunc type alias.

Type alias of a function doesn't really add any benefit other than documentation, since no additional methods will be added to the type alias.

func main() {
	var fn MyFunc

	fn = Impl
	fmt.Println(fn(&SomeType{}))

	SomeConfig(SomeSetup()...)
}

type MyFunc func(*SomeType) error

func Impl(t *SomeType) error {
	return nil
}

func SomeConfig(fns ...MyFunc) {

}

type SomeType struct{}

func SomeSetup() []func(*SomeType) error {

	return []func(*SomeType) error{}
}
cannot use SomeSetup() (type []func(*SomeType) error) as type []MyFunc in argument to SomeConfig

https://play.golang.org/p/754BI_LsS_6

Operation cloned Input/Output shapes can collide with types shape of same name

The (de)serializer codegen looks like it can collide cloned operation input/output shapes with types of the same name used nested within the shape tree.

./test_submodules.sh service/glacier 'go test -run NONE'
Testing github.com/aws/aws-sdk-go-v2/service/glacier
# github.com/aws/aws-sdk-go-v2/service/glacier [github.com/aws/aws-sdk-go-v2/service/glacier.test]
./deserializers.go:1443:60: cannot use &output (type **DescribeVaultOutput) as type **types.DescribeVaultOutput in argument to awsRestjson1_deserializeDocumentDescribeVaultOutput
./deserializers.go:5354:6: awsRestjson1_deserializeDocumentDescribeVaultOutput redeclared in this block
	previous declaration at ./deserializers.go:1519:106
FAIL	github.com/aws/aws-sdk-go-v2/service/glacier [build failed]

Incorrect time formatting

FormatDateTime is unsafe!

It uses the formatting string dateTimeFormat = "2006-01-02T15:04:05.99Z" which essentially just appends the Z character onto the user's local time.

Possible fixes:

  • Swap the format string to "2006-01-02T15:04:05.99Z07:00" or maybe even time.RFC3339 (probably breaking)
  • Explicitly cast to UTC time (maybe breaking - but probably just bug fixing)
func FormatDateTime(value time.Time) string {
      return value.UTC().Format(dateTimeFormat)
 }
  • Document that the function should be called with a UTC time (probably too conservative)

Not sure what your original intentions were so I'll leave this to you.

I originally found this after spending an hour or so debugging why calls to cloudwatch.GetMetricData weren't returning any data in the v2 aws SDK.

http body need close. Otherwise, it may cause goroutine leakage in the process

// Body represents the response body.
//
// The response body is streamed on demand as the Body field
// is read. If the network connection fails or the server
// terminates the response, Body.Read calls return an error.
//
// The http Client and Transport guarantee that Body is always
// non-nil, even on responses without a body or responses with
// a zero-length body. It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.
//
// The Body is automatically dechunked if the server replied
// with a "chunked" Transfer-Encoding.
//
// As of Go 1.12, the Body will also implement io.Writer
// on a successful "101 Switching Protocols" response,
// as used by WebSockets and HTTP/2's "h2c" mode.
Body io.ReadCloser

image

`internaldocument.NewDocumentMarshaler` not declared by package

Hi,

I generated Go types from smithy models. One member shape uses the Document type. The generated code doesn't seem to compile:

In file: codegen/document/document.go:

func NewLazyDocument(v interface{}) Interface {
	return internaldocument.NewDocumentMarshaler(v)
}

I get an error: NewDocumentMarshaler not declared by package document

Any ideas?

GenerateStandaloneGoModuleTest: Test breaks if in a directory hierarchey containing `go.work` file.

var testPath = getTestOutputDir();
LOGGER.warning("generating test suites into " + testPath);
var fileManifest = FileManifest.create(testPath);
var writers = new GoWriterDelegator(fileManifest);

Suggestion: The test files should be written to a temporary file directory created using something like Files.createTempDirectory so we can try to get a clean directory that would more then likely fall outside a file system directory with a go.work file present in the hierarchy.

API Client's operation methods can be simplified by reusing behavior in helper function

I think we could simplify the client operation method generation a bit by generating a helper method on the API client type that is used by all of the operations reducing a large amount of the duplicate code.

func (c *Client) invokeOperation(
	ctx context.Context, opID string, params interface{}, optFns []func(*Options),
	stackFns ...func(*middleware.Stack) error,
) (
	result interface{}, metadata middleware.Metadata, err error,
) {
	stack := middleware.NewStack(opID, smithyhttp.NewStackRequest)
	options := c.options.Copy()
	for _, fn := range optFns {
		fn(&options)
	}

	for _, fn := range stackFns {
		if err := fn(stack); err != nil {
			return nil, metadata, err
		}
	}

	for _, fn := range options.APIOptions {
		if err := fn(stack); err != nil {
			return nil, metadata, err
		}
	}

	handler := middleware.DecorateHandler(smithyhttp.NewClientHandler(options.HTTPClient), stack)
	result, metadata, err = handler.Handle(ctx, params)
	if err != nil {
		return nil, metadata, &smithy.OperationError{
			ServiceID:     ServiceID,
			OperationName: opID,
			Err:           err,
		}
	}

	return result, metadata, err
}

Would be used by the client's operation method, to do all of the common behavior.

// GetInstanceIdentityDocument retrieves an identity document describing an
// instance. Error is returned if the request fails or is unable to parse
// the response.
func (c *Client) GetInstanceIdentityDocument(
	ctx context.Context, params *GetInstanceIdentityDocumentInput, optFns ...func(*Options),
) (
	*GetInstanceIdentityDocumentOutput, error,
) {
	if params == nil {
		params = &GetInstanceIdentityDocumentInput{}
	}

	result, metadata, err := c.invokeOperation(ctx, "GetInstanceIdentityDocument", params, optFns,
		addGetDynamicDataMiddleware,
                moreMiddleare,
                moreOtherMiddleare,
	)

	out := result.(*GetInstanceIdentityDocumentOutput)
	out.ResultMetadata = metadata
	return out, nil
}

Need help in using Smithy Go

Hello! We at Grafana Labs are trying to prototype HTTP API client and server Go code generation, with the help of Smithy and the Smithy Go plugin. The documentation is very sparse though, and we're not having much luck yet. I've basically got to the point of being able to have the smithy executable load the Go plugin, but then it fails:

$ cat smithy-build.json
{
  "version": "1.0",
  "plugins": {
    "software.amazon.smithy.go.codegen.GoCodegenPlugin": {}
  }
}
$ ~/Projects/grafana/smithy/smithy-cli/build/image/bin/smithy build
Building Smithy model sources: []
Exception in thread "main" java.lang.NoClassDefFoundError: software/amazon/smithy/build/model/SmithyBuildConfig
	at software.amazon.smithy.cli.commands.BuildCommand.execute(BuildCommand.java:82)
	at software.amazon.smithy.cli.Cli.run(Cli.java:144)
	at software.amazon.smithy.cli.SmithyCli.run(SmithyCli.java:93)
	at software.amazon.smithy.cli.SmithyCli.main(SmithyCli.java:53)
Caused by: java.lang.ClassNotFoundException: software.amazon.smithy.build.model.SmithyBuildConfig
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	... 4 more

I'm on OSX, using OpenJDK (version 8 to build Smithy Go, and version 15 to build Smithy itself since v8 wouldn't work). Could someone please provide guidance in getting Go code generation working with Smithy?

Middleware steps should support a slotting.

Middleware steps should allow for a slot to be added to a given step via an AddSlot and InsertSlot API. A slot would be a named position with the middleware step that would allow other middlewares to be inserted relative to it. If a middleware is registered with a matching slot name, that middleware would fill in the slot. If a middleware is later removed, the slot reference would still remain.

Slots that are not directly assigned a middleware would have no affect on the fully constructed handler stack.

Group encoding packages under common namespace to improve shared logic

It would be helpful if the encoding packages like json, xml, query, etc, where grouped under a common encoding pkg. This makes it easier for these packages to have a common internal pkg containing shared behavior such as encodeFloat.

Propose a pkg refactor similar to:

/smithy-go
     /encoding
          /json
          /xml
          /query
          /httpbinding (aka rest)
          /eventstream
          /...
          /internal
               /float.go
               /...
      /transport
          /http
          /mqtt
          /eventstream
          /...

API clients send Content-Type even when Content-Length is empty

The generated API clients should remove the Content-Type request header when the request body is empty, (e.g. Content-Length=0) for structured payloads. The API clients probably should still send Content-Type for operations with streaming inputs.

Protocol test should also be created for this use case for the REST-JSON protocol, and any other protocol that can serialize empty request body for structured payloads.

API clients should be generated with a WithAPIOptions helper function

Each API client should be generated with a WithAPIOptions function. This function allows the user to specify API options without the need for creating function closures in their code.

Generate the helper into the API client's package.

func WithAPIOptions(optFns ...func(*middleware.Stack) error) func (*Options)

Proposed usage example:

client := dynamodb.NewFromConfig(cfg, dynamodb.WithAPIOptions(
      middleware.WithUserAgentKeyValue("foo", "bar"),
))

Current design's usage:

client := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
      o.APIOptions = append(o.APIOptions, middleware.WithUserAgentKeyValue("foo", "bar"))
})

Also depends on: #144

Client options must not generate Getter style methods for retrievng members

This issue tracks the removal of the generated GetXYZ() XYZ methods from the client Options type. Getters are an anti-pattern that has quickly broken down when attempting to define "Getters" around members that are interface types. Specifically if a client attempts to decouple itself from an implementation by defining its own interface definition for an option member, the generated "Getter" would be unable to satisfy an interface that describes the same Getter but with an alternative (but matching) interface type.

As an example the following code would not allow the client.Options{} to be passed into fooimpl.NewFooMiddleware

package client

type FooInterface interface {
  DoThing() bool
}

type Options struct {
  Foo FooInterface
}

func (o *Options) GetFoo() FooInterface {
  return o.Foo
}
...
package fooimpl

type Foo interface {
  DoThing() bool
}

type FooImpl struct {}
func (f *FooImpl) DoThing() {
  return true
}

type FooMiddlewareOptions interface {
  GetFoo() Foo
}

func NewFooMiddleware(o Options) FooMiddleware {
  ...
}

Generated validators should include Union shapes

The generated validators do not have support for union shapes. Required (not nil), and nested member required validation should be added. Potentially generate a helper for the union shape container to make it easier to validate individual union members.

This validation probably should also assert that the union member value is not a UnknownUnionMember type.

https://github.com/awslabs/smithy-go/blob/660ac76fa00a13b3a912e1613e70984401cdb6c1/codegen/smithy-go-codegen/src/main/java/software/amazon/smithy/go/codegen/integration/ValidationGenerator.java#L203-L204

Alias type name hides interface type name in generated input/output types

Alias type name hides interface function name.

This led me down an interesting exploration of go and how a field can hide a function given by an anonymous struct field.

It makes the code confusing to someone reading it. What is the benefit of naming these the same thing? If there's no practical benefit - why not have the interface method and the type alias have different names? The method isn't exported anyway and won't clutter the autocomplete of customers.

GoCodeWriter's $T and $P generate unexpected types for list and maps.

We should take a look at how slice/map types are generated from $T, and $P. If a member is or is not a pointer of a slice/map should not be driven by $T, or $P, the member being a pointer is driven by the modeled list/map shape, and its member.

This mean that $T should generate the correct Go type for a map of strings, i.e. map[string]*string. In addition the same for lists (assuming lists cannot be sparse, i.e. []string. Using $P should generate the pointer type of the outer shape, *map[string]*string and *[]string. $P should have no impact on the generate of nested members within collections.

smithy-go does not provide support for @restJson1 protocol trait

When generating Go code, I get this error:

Unable to find a protocol generator for my.service#MyService: The my.service#MyService service supports the following unsupported protocols [aws.protocols#restJson1]. The following protocol generators were found on the class path: []

I opened this issue: smithy-lang/smithy#1101 with the smithy repo, but I was redirected here.

The solution proposed was to:

The actual code generation for the restJson1 protocol, because it's an AWS protocol, comes from software.amazon.smithy:smithy-aws-go-codegen... which isn't published to Maven Central. Can you open this ticket on the smithy-go repo so they can either a) move the restJson1 codegen to smithy-go directly, or b) publish smithy-aws-go-codegen to Maven Central?

enhancement: Option to generate a go.mod file

We have a folder structure like this:

.
├── model
│   ├── build/smithyprojections/model/source/go-codegen
└── externalLib

It's a monorepo where externalLib has a go.mod file.

require github.com/sharedLib v0.0.0

replace github.com/sharedLib => ../model/build/smithyprojections/model/source/go-codegen

Maybe a PEBKAC problem, but the import wouldn't work unless we went into the go-codegen/ and manually created a go.mod file.

If this makes sense to add, I'd be happy to make a pull request if you could point me in the general direction. My thinking is either do it by default or by adding an opt-in option in smithy-build.json

`document.NoSerde` is not checked for nesteted types.

When marshalling a Go type to a document type it was discovered that the document.IsNoSerde check to determine whether a type is a generated Go type defined in the Smithy model only validates the passed in shape. In this updated test snippet below, it is shown that a custom user defined Go type using a generated type as a field does not correctly prevent usage of the generated type.

func TestNewEncoderUnsupportedTypes(t *testing.T) {
	type customTime time.Time
	type noSerde = document.NoSerde
	type NestedThing struct {
		SomeThing string
		noSerde
	}
	type Thing struct {
		OtherThing  string
		NestedThing NestedThing
	}

	cases := []interface{}{
		time.Now().UTC(),
		customTime(time.Now().UTC()),
		Thing{OtherThing: "cool", NestedThing: NestedThing{SomeThing: "thing"}},
	}

	encoder := json.NewEncoder()
	for _, tt := range cases {
		_, err := encoder.Encode(tt)
		if err == nil {
			t.Errorf("expect error, got nil")
		}
	}
}

IsNoSerde check should be validated for any type that is nested. So usages in fields, or as slice types, or map types etc.

Support renamed errors in aws-json-1.1 protocol

Describe the feature

Smithy recently allowed renaming error shapes but added a validator to prevent renamed errors with aws protocols.

Most aws protocols only use shape name in error serialization, which is why error renames were originally banned. However, aws json 1.0 uses qualified shape id and can support renamed errors.

The protocol should match errors based on the original (not renamed) fully qualified shape id.

Once all smithy sdk's (rust, js, go, kotlin) confirm that aws json 1.0 supports renamed errors smithy can remove the protocol from the validator and allow renames.

smithy-lang/smithy#1545

Use Case

I have smithy a smithy model with shapes with conflicting names, tho the qualified shape id is unique. I'd like to use aws json 1.0 protocol, which could support renamed error shapes, but the validator throws an error.

Proposed Solution

No response

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

AWS Go SDK V2 Module Versions Used

N/A

Go version used

N/A

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.