utrack / clay Goto Github PK

View Code? Open in Web Editor NEW

288.0 288.0 39.0 2.75 MB

Proto-first minimal server platform for gRPС+REST+Swagger APIs

License: MIT License

Go 80.62% Makefile 19.17% Shell 0.21%

golang grpc grpc-gateway protobuf swagger

clay's People

Contributors

Stargazers

Watchers

clay's Issues

Add support of flag 'module' to 'goclay_out'

The current implementation of protoc-gen-go supports module flag, that (and that flag is supported even in new APIv2 since v1.21.0). The current implementation of protoc-gen-goclay plugin has no support for module flag:

Error parsing flags: Cannot set flag module=github.com/foo/bar: no such flag -module

Using this flag helps to enhance workflow with generated files due to generation in the root directory with a relative path (i.e. github.com/foo/bar/baz/my.proto will be generated as baz/my.pb.go with --go_out=module=github.com/foo/bar).

With the support of this flag, it'll be much easier to work in modules and overall with generated files.

Define errors' model for Swagger docs

Since grpc-ecosystem/grpc-gateway#1127 it's possible to define an Error message via registering .grpc.gateway.runtime.Error and .grpc.gateway.runtime.StreamError.

Right now, these definitions are explicitly disabled in clay via SetDisableDefaultErrors(true), but it would be great to reenable them if the user somehow provides an Error when generating Go code.

interface conversion: interface {} is nil, not map[string]interface {}

transport.NewCompoundServiceDesc().SwaggerDef()
return:
{"definitions":null,"paths":null}
expect:
{"definitions":{},"paths":{}}
or not panic in example

s1 := transport.NewCompoundServiceDesc()
s2 := transport.NewCompoundServiceDesc(s1)

s2.SwaggerDef() // <-- panic

I think you shouldn't call type cast before nil check
https://github.com/utrack/clay/blob/master/transport/swagcombo.go#L30-L36

How to implement emit default to response or have flag option

https://stackoverflow.com/questions/34716238/golang-protobuf-remove-omitempty-tag-from-generated-json-tags
The zero vaule of field be omited, I remove omitempty flag in struct, but it do not work in reponse

Generate HTTP client with a server

Apply a configurable swagger option

I think these options (d.opts.SwaggerDefaultOpts) should be present in this condition

If a service isConfigurableServiceDesc, we can append additional swagger option but not check it

proto gen: support optionals in proto syntax 3

Anytime I use this plugin to generate code from a proto file, I get this error message:

example.proto is a proto3 file that contains optional fields, but code generator protoc-gen-goclay hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.--goclay_out:

Since this update supports optionals by default, we should support optionals during code generation?

Support full go package paths in `option go_package`

See golang/protobuf#139

At the moment, generator doesn't create subdirs for the generated .goclay and .swagger file, putting it in the root output folder.

Support sending response headers via `grpc.SetHeader`

genhandler: create own registry

ref #97, preparation for v3

Currently, genhandler is closely tied up to protobuf & grpc-gateway structures.

Instead of using external abstractions and calling functions from the template, we should build our own registry that'll already have all the data we would need to generate that template. This will make it easier to change templates.

See this example: https://github.com/utrack/clay/blob/master/cmd/protoc-gen-goclay/genhandler/tpl_servicedesc.go#L8
It'll be easier if every Service would already have .GoTypeName field embedded in it.

This map contains all the functions that are called from the template: https://github.com/utrack/clay/blob/master/cmd/protoc-gen-goclay/genhandler/template.go#L224

template.go:1056] no message found: Error

From time to time while using goclay I see these messages in shell:

template.go:1056] no message found: Error

In this situation an extra code for swagger is being generated:

"default": {
  "description": "An unexpected error response",
  "schema": {
    "$ref": "#/definitions/"
  }
}

Which leads to issues on swagger-ui with outputting resources.

As far as I understood, it happens when .pb.goclay.go is already exist.

grpc-gateway can run inside the same process

Really nifty project. Thanks for open sourcing! This looks like a great resource for the community.

I was just looking through the code and it seems I have done a poor job of documenting gRPC-gateways' ability to serve from inside the same process. In our example grpc server we allow you to pass in a network which could be a "tcp", "tcp4" (IPv4-only), "tcp6" (IPv6-only), "udp", "udp4" (IPv4-only), "udp6" (IPv6-only), "ip", "ip4" (IPv4-only), "ip6" (IPv6-only), "unix", "unixgram" and "unixpacket". All of the unix class ones would take a system path as "addr" and let you serve all traffic without opening a TCP connection.

How could I help make that more clear?

HTTP client: path builder uses wrong names for nested bindings in path

service Strings {
  rpc ToUpper(String) returns (Empty) {
    option (google.api.http) = {
      put : "/strings/{sub.id}"
      body : "sub"
      additional_bindings {patch : "/strings/{sub.id}" body : "sub"}
    };
  }
}

HTTP query builder uses proto member names instead of Go names:

u := url.URL{
	Path:     fmt.Sprintf("/strings/%v", in.sub.Id),
	RawQuery: values.Encode(),
}

Move to protoreflect

Move to https://github.com/jhump/protoreflect because current registry is severely limited because of its original scope.

Unmarshal panic

https://github.com/utrack/clay/blob/master/transport/httpruntime/mjson.go#L41
it's not correct logical operator && for v = v.Elem() invoked here

https://golang.org/pkg/reflect/#Value.Elem
It panics if v's Kind is not Interface OR Ptr.

I think it should be ||

Forward HTTP headers to the handler via gRPC metadata

Currently HTTP-gRPC headers are passed if user supplies the proper HTTP middleware; we need to do it in the generated code.

Check if we're trying to generate services for included protos

We shouldn't generate services for protofiles that are included from elsewhere.

Generated code header doesn't conform to the standard

golang/go#13560 (comment)

Our generated code header is wrong

// Code generated by protoc-gen-goclay
// source: sum.proto
// DO NOT EDIT!

DO NOT EDIT should be on the first line.

Arrays of objects are not allowed in http url query parameters

For example an RPC like this

service MapService {
  rpc Search(SearchRequest) returns (SearchResponse) {
    option (google.api.http) = {
      get: "/v1/search"
    };
  }
}

message SearchRequest{
  repeated Coordinate polygon = 1;
}

message Coordinate {
  float lat = 1;
  float lon = 2;
}

is expected to be callable using http protocol in the following manner: https://example.com/v1/search?polygon[0][lat]=55.11&polygon[0][lon]=55.22 but at the moment fields with repeated custom messages are just ignored from generated http server and swagger spec.

Support field_mask for PATCH requests

Re #72

FieldMask is unsupported right now, generated code should hook into grpc-gateway FieldMask factories.
docs for the feature: https://grpc-ecosystem.github.io/grpc-gateway/docs/patch.html

Swagger definition does not have comments for path params

panic reflect: call of reflect.Value.Type on zero Value when sending patch request.

I tried setting up a patch request gateway base from grpc_gateway documentation here.

Basically, I'm using field_mask and I'm trying to use the built-in behavior of grpc_gateway so that the client of the endpoint don't need to set the field_mask manually.

here is the callstack:

http: panic serving [::1]:58216: reflect: call of reflect.Value.Type on zero Value
goroutine 73 [running]:
net/http.(*conn).serve.func1(0xc001242000)
        /usr/local/Cellar/go/1.13.4/libexec/src/net/http/server.go:1767 +0x139
panic(0x19ae0e0, 0xc000252d40)
        /usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
reflect.Value.Type(0x0, 0x0, 0x0, 0x1a30300, 0x400)
        /usr/local/Cellar/go/1.13.4/libexec/src/reflect/value.go:1877 +0x166
github.com/golang/protobuf/jsonpb.(*Unmarshaler).unmarshalValue(0xc000252d00, 0x0, 0x0, 0x0, 0xc000039a60, 0x11, 0x20, 0x0, 0x100ebd8, 0x20)
        /Users/user/Workspace/go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:720 +0x5d
github.com/golang/protobuf/jsonpb.(*Unmarshaler).UnmarshalNext(0xc000252d00, 0xc0011ac2c0, 0x1d54d00, 0x0, 0x0, 0x19a0101)
        /Users/user/Workspace/go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:682 +0x166
github.com/grpc-ecosystem/grpc-gateway/runtime.decodeJSONPb(0xc0011ac2c0, 0x1acc1c0, 0x0, 0x1d6ef01, 0xc0011ac2c0)
        /Users/user/Workspace/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:176 +0xa5
github.com/grpc-ecosystem/grpc-gateway/runtime.DecoderWrapper.Decode(0xc0011ac2c0, 0x1acc1c0, 0x0, 0x1d54d00, 0x0)
        /Users/user/Workspace/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:149 +0x3f
github.com/utrack/clay/v2/transport/httpruntime.MarshalerPbJSON.Unmarshal(0x23e5380, 0x23e53c0, 0x23e5400, 0x23e34e0, 0x73024d8, 0xc000559fc0, 0x1acc1c0, 0x0, 0x1d59800, 0x73024d8)
        /Users/user/Workspace/go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/mjson.go:59 +0x25f
github.com/org/project/pkg/transaction_service.glob..func11.1(0x1a6f620, 0xc0004100c0, 0xc00125b788, 0x100ebd8)
        /Users/user/Workspace/go/src/github.com/org/project/pkg/transaction_service/transaction.pb.goclay.go:802 +0x168
github.com/org/project/pkg/transaction_service._TransactionService_UpdateTransaction_Handler(0x1a1c9a0, 0xc000520030, 0x1d5c640, 0xc00029f680, 0xc000276f20, 0xc000459590, 0x30, 0x30, 0x1a53f40, 0xc00125b8a0)
        /Users/user/Workspace/go/src/github.com/org/project/pkg/transaction_service/transaction.pb.go:1956 +0x53
github.com/org/project/pkg/transaction_service.(*TransactionServiceDesc).RegisterHTTP.func7(0x1d57b40, 0xc000410080, 0xc0001c6500)
        /Users/user/Workspace/go/src/github.com/org/project/pkg/transaction_service/transaction.pb.goclay.go:289 +0x102
net/http.HandlerFunc.ServeHTTP(...)

Any help would be appreciated. Thanks.

How to setting allow_delete_body option is true？

I want to allow delete request have a request body , but I don't konw how to setting ,
I use yaml to config my http define ,

--goclay_out: must not set request body when http method is DELETE except allow_delete_body option is true: DeleteUser

http:
  rules:
  -  selector: user_center.ManageUserCenter.DeleteUser
      delete: '/v1/example/userCenter/deleteUser'
      body: 'data'
      allow_delete_body: true

it does not work

Upload file via API

Create new httpruntime.Marshaler that supports multipart messages and octet-stream.

Unbound parameters are ignored in query paths

From this proto:

message ListCreativesRequest {
    uint64 marketing_action_id = 1;
}

service CreativeService {
    rpc ListCreative (ListCreativesRequest) returns (ListCreativeResponse) {
        option (google.api.http) = {
    	    get: "/v1/creatives"
    	};
    }
}

will be generated wrong unmarshaller that doesn't fill ListCreativesRequest struct.

pattern_goclay_CreativeService_ListCreative_0     = "/v1/creatives"
unmarshaler_goclay_CreativeService_ListCreative_0 = func(r *http.Request, req *ListCreativesRequest) error {

	return nil

}

Unmarshal http request body to specified field

Sample proto:

message UpdateCreativeRequest {
    uint64 id = 1;
    Creative creative = 2;
}

service CreativeService {
    rpc UpdateCreative (UpdateCreativeRequest) returns (Creative) {
        option (google.api.http) = {
    	    put: "/v1/creatives/{id}"
    	    body: "creative"
    	};
    }
}

Version 3

With the protoc-gen-go v1.20.0+ and new gRPC generator breaking compatibility (see grpc/grpc-go#3828), it seems that a big refactoring process will be needed (and frankly, current code calls for it).

This is a good chance to implement some things in a better way and throw away some others: for example, tricky output rules pre---module flag.

This is an umbrella issue that will collect everything that will go to v3.0.0.

Get rid of reflect hack for gogo/protobuf

#33 might fix it since we've hit the limit on the current gateway registry, or wait until golang and gogo converge on a plan for their big refactoring.
It should be possible to crutch-fix that via reflection in the JSONMarshaler meanwhile.

Does not correctly generate pb.goclay.go with filed option nullable = false

Clay does not correct generate .pb.goclay.go file, if in proto-file exists into body exists one of proto-field with [(gogoproto.nullable) = false];

Example proto-file

rpc UserV1 (UserRequest) returns (UserResponse) {
        option (google.api.http) = {
            post: "UserV1"
            body: "user"
        };
    }

...
message UserRequest {
    User user = 1; [(gogoproto.nullable) = false];
}

Example piece generated code into .pb.goclay.go

...
req.User = &User{}
...

Error into generate template - https://github.com/utrack/clay/blob/master/cmd/protoc-gen-goclay/genhandler/template.go#L310

No chi CloseNotify middleware

In github.com/utrack/clay/server/middlewares/mwhttp/closenotifier.go github.com/go-chi/chi/middleware.CloseNotify is not available now. It's deleted from go-chi/chi

protobuf/protoc-gen-go/generator is deprecated

The package is deprecated since 1.4.0 (stable release 2020-04-16)
https://pkg.go.dev/github.com/golang/[email protected]/protoc-gen-go/generator?tab=doc

Implement Validator interface

We need add Validator interface for Request struct. When request struct fetch data from request and if it implement Validator interface, we can validate data in request and return error before handler.

type Validator interface {
	Valid() bool
}

Problem with folder names containing dash when use prototool

I use prototool to call generation and got error:

$ prototool all "pb/" 
could not interpret protoc line: E0615 15:20:09.519821   23865 handler.go:72] 114:18: expected ';', found '-': 0:

    .
    ├── ...
    ├── pb                         
    │   └── hello-world 
    │       └── hw.proto 
    └── ...

Proto type int64 get the error format in http swagger post body

proto file

message Userprofile{
  int64 user_id = 1;
  ...
 }

pb.go

type Userprofile struct {
	UserId int64 `protobuf:"varint,1,opt,name=user_id,json=userId,proto3" json:"user_id,omitempty"`
  ....
}

${8CE3E2BC-BF98-4D2F-BC36-BFE9087C216E}_20200616155100$
It’s very strange ~, but I try to use int32 , it's work fine

go.mod have two logger system

github.com/sirupsen/logrus v1.4.2 and
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b

Get The reflect error in example

I browse swagger ui and input a ,b ,I get this error ,

oot@9de65925afcf:/workspaces/example# go run main.go 
go: finding module for package github.com/utrack/clay/v2/transport/httptransport
go: finding module for package github.com/utrack/clay/v2/transport/httpclient
go: found github.com/utrack/clay/v2/transport/httpclient in github.com/utrack/clay/v2 v2.4.7
2020/06/05 16:26:12 http: panic serving 127.0.0.1:59764: reflect: call of reflect.Value.Type on zero Value
goroutine 51 [running]:
net/http.(*conn).serve.func1(0xc00017e000)
        /usr/local/go/src/net/http/server.go:1772 +0x139
panic(0xa01be0, 0xc0003ea5e0)
        /usr/local/go/src/runtime/panic.go:975 +0x3e3
reflect.Value.Type(0x0, 0x0, 0x0, 0x30, 0xa95320)
        /usr/local/go/src/reflect/value.go:1872 +0x183
github.com/golang/protobuf/jsonpb.(*Unmarshaler).unmarshalValue(0xc0003ea560, 0x0, 0x0, 0x0, 0xc0003ca890, 0x10, 0x10, 0x0, 0x40e478, 0x20)
        /go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:720 +0x5d
github.com/golang/protobuf/jsonpb.(*Unmarshaler).UnmarshalNext(0xc0003ea560, 0xc0000ec6e0, 0xca8260, 0x0, 0x0, 0xa08301)
        /go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:682 +0x166
github.com/grpc-ecosystem/grpc-gateway/runtime.decodeJSONPb(0xc0000ec6e0, 0xa7a880, 0x0, 0xcb4a01, 0xc0000ec6e0)
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:176 +0xa5
github.com/grpc-ecosystem/grpc-gateway/runtime.DecoderWrapper.Decode(0xc0000ec6e0, 0xa7a880, 0x0, 0xca8260, 0x0)
        /go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:149 +0x3f
github.com/utrack/clay/v2/transport/httpruntime.MarshalerPbJSON.Unmarshal(0x10db300, 0x10db340, 0x10db380, 0x10d9c00, 0x7f0c15b30d68, 0xc0000b9340, 0xa7a880, 0x0, 0xca9200, 0x7f0c15b30d68)
        /go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/mjson.go:59 +0x25f
github.com/utrack/clay/doc/example/pb.glob..func2.1(0xa81180, 0xc0003b09f0, 0xc0001cb788, 0x40e478)
        /workspaces/example/pb/sum.pb.goclay.go:234 +0x2a9
github.com/utrack/clay/doc/example/pb._Summator_Sum_Handler(0xa12880, 0x1106250, 0xcaa4a0, 0xc0003b09c0, 0xc0003a8440, 0xc0003b0060, 0x80, 0x80, 0xa950a0, 0xc000080000)
        /workspaces/example/pb/sum.pb.go:209 +0x53
github.com/utrack/clay/doc/example/pb.(*SummatorDesc).RegisterHTTP.func1(0xca8360, 0xc0000b93c0, 0xc0000c0600)
        /workspaces/example/pb/sum.pb.goclay.go:108 +0xe8
net/http.HandlerFunc.ServeHTTP(...)
        /usr/local/go/src/net/http/server.go:2012
github.com/utrack/clay/v2/transport/httpruntime/httpmw.HeadersToGRPCMD.func1(0xca8360, 0xc0000b93c0, 0xc0000c0500)
        /go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/httpmw/headers.go:45 +0x3d1
net/http.HandlerFunc.ServeHTTP(...)
        /usr/local/go/src/net/http/server.go:2012
github.com/utrack/clay/v2/transport/httpruntime/httpmw.InjectTransportStream.func1(0xca8be0, 0xc0000b2380, 0xc0000c0400)
        /go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/httpmw/headers.go:57 +0x23b
net/http.HandlerFunc.ServeHTTP(0xc0003a8140, 0xca8be0, 0xc0000b2380, 0xc0000c0400)
        /usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0000c26c0, 0xca8be0, 0xc0000b2380, 0xc0000c0400)
        /go/pkg/mod/github.com/go-chi/[email protected]+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc0003a80e0, 0xca8be0, 0xc0000b2380, 0xc0000c0400)
        /usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0000c26c0, 0xca8be0, 0xc0000b2380, 0xc00017c400)
        /go/pkg/mod/github.com/go-chi/[email protected]+incompatible/mux.go:81 +0x2b2
net/http.serverHandler.ServeHTTP(0xc0000b20e0, 0xca8be0, 0xc0000b2380, 0xc00017c400)
        /usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc00017e000, 0xcaa3e0, 0xc000068000)
        /usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2933 +0x35c

grpc client run correctly

Swagger joiner loose parameters after SumDefinitions

You should store key "parameters" as well as "definitions"

Looking for new maintainers / possible project deprecation

Putting up this issue to ping everyone who watches this repo.

See the announcement here: ff09f40

Ability to add custom swagger options

Need a way to add my custom options to swagger.json.
I need this to add auth headers that I use.

Option to suppress non-fatal warnings

We need to add an option to suppress non-fatal warnings to make clay work w/ prototool (see uber/prototool#128).

Add emitting default values support in JSON

It will be great if I can paste some flag to generator which pass EmitDefaults value in default marshaler. This option is well discussed in that thread: grpc-ecosystem/grpc-gateway#233
I suppose to make it optional with false as default value not to change current behavior.
I'm talking about flag exactly, not to develop some custom httpruntime.Marshaler implementation and injection... If @utrack will be agree, I'll implement this feature.

Wrong GoPkg used with .Method.RequestType.GoType inside tpl generation

There is a mistake with deferring GoPkg after getImplParam implementation (getImplTemplate refactoring) - no relative GoPkg got from getImplParam function

utrack / clay Goto Github PK

clay's People

Contributors

Stargazers

Watchers

Forkers

clay's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs