utrack / clay Goto Github PK
View Code? Open in Web Editor NEWProto-first minimal server platform for gRPС+REST+Swagger APIs
License: MIT License
Proto-first minimal server platform for gRPС+REST+Swagger APIs
License: MIT License
The current implementation of protoc-gen-go
supports module
flag, that (and that flag is supported even in new APIv2 since v1.21.0
). The current implementation of protoc-gen-goclay
plugin has no support for module
flag:
Error parsing flags: Cannot set flag module=github.com/foo/bar: no such flag -module
Using this flag helps to enhance workflow with generated files due to generation in the root directory with a relative path (i.e. github.com/foo/bar/baz/my.proto
will be generated as baz/my.pb.go
with --go_out=module=github.com/foo/bar
).
With the support of this flag, it'll be much easier to work in modules and overall with generated files.
Since grpc-ecosystem/grpc-gateway#1127 it's possible to define an Error message via registering .grpc.gateway.runtime.Error
and .grpc.gateway.runtime.StreamError
.
Right now, these definitions are explicitly disabled in clay via SetDisableDefaultErrors(true)
, but it would be great to reenable them if the user somehow provides an Error
when generating Go code.
transport.NewCompoundServiceDesc().SwaggerDef()
return:
{"definitions":null,"paths":null}
expect:
{"definitions":{},"paths":{}}
or not panic in example
s1 := transport.NewCompoundServiceDesc()
s2 := transport.NewCompoundServiceDesc(s1)
s2.SwaggerDef() // <-- panic
I think you shouldn't call type cast before nil
check
https://github.com/utrack/clay/blob/master/transport/swagcombo.go#L30-L36
https://stackoverflow.com/questions/34716238/golang-protobuf-remove-omitempty-tag-from-generated-json-tags
The zero vaule of field be omited, I remove omitempty flag in struct, but it do not work in reponse
Anytime I use this plugin to generate code from a proto file, I get this error message:
example.proto is a proto3 file that contains optional fields, but code generator protoc-gen-goclay hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.--goclay_out:
Since this update supports optionals by default, we should support optionals during code generation?
At the moment, generator doesn't create subdirs for the generated .goclay
and .swagger
file, putting it in the root output folder.
ref #97, preparation for v3
Currently, genhandler
is closely tied up to protobuf & grpc-gateway structures.
Instead of using external abstractions and calling functions from the template, we should build our own registry that'll already have all the data we would need to generate that template. This will make it easier to change templates.
See this example: https://github.com/utrack/clay/blob/master/cmd/protoc-gen-goclay/genhandler/tpl_servicedesc.go#L8
It'll be easier if every Service would already have .GoTypeName
field embedded in it.
This map contains all the functions that are called from the template: https://github.com/utrack/clay/blob/master/cmd/protoc-gen-goclay/genhandler/template.go#L224
From time to time while using goclay I see these messages in shell:
template.go:1056] no message found: Error
In this situation an extra code for swagger is being generated:
"default": {
"description": "An unexpected error response",
"schema": {
"$ref": "#/definitions/"
}
}
Which leads to issues on swagger-ui with outputting resources.
As far as I understood, it happens when .pb.goclay.go
is already exist.
Really nifty project. Thanks for open sourcing! This looks like a great resource for the community.
I was just looking through the code and it seems I have done a poor job of documenting gRPC-gateways' ability to serve from inside the same process. In our example grpc server we allow you to pass in a network
which could be a "tcp", "tcp4" (IPv4-only), "tcp6" (IPv6-only), "udp", "udp4" (IPv4-only), "udp6" (IPv6-only), "ip", "ip4" (IPv4-only), "ip6" (IPv6-only), "unix", "unixgram" and "unixpacket". All of the unix class ones would take a system path as "addr" and let you serve all traffic without opening a TCP connection.
How could I help make that more clear?
service Strings {
rpc ToUpper(String) returns (Empty) {
option (google.api.http) = {
put : "/strings/{sub.id}"
body : "sub"
additional_bindings {patch : "/strings/{sub.id}" body : "sub"}
};
}
}
HTTP query builder uses proto member names instead of Go names:
u := url.URL{
Path: fmt.Sprintf("/strings/%v", in.sub.Id),
RawQuery: values.Encode(),
}
Move to https://github.com/jhump/protoreflect because current registry is severely limited because of its original scope.
https://github.com/utrack/clay/blob/master/transport/httpruntime/mjson.go#L41
it's not correct logical operator &&
for v = v.Elem()
invoked here
https://golang.org/pkg/reflect/#Value.Elem
It panics if v's Kind is not Interface OR Ptr.
I think it should be ||
Currently HTTP-gRPC headers are passed if user supplies the proper HTTP middleware; we need to do it in the generated code.
We shouldn't generate services for protofiles that are included from elsewhere.
Our generated code header is wrong
// Code generated by protoc-gen-goclay
// source: sum.proto
// DO NOT EDIT!
DO NOT EDIT
should be on the first line.
For example an RPC like this
service MapService {
rpc Search(SearchRequest) returns (SearchResponse) {
option (google.api.http) = {
get: "/v1/search"
};
}
}
message SearchRequest{
repeated Coordinate polygon = 1;
}
message Coordinate {
float lat = 1;
float lon = 2;
}
is expected to be callable using http protocol in the following manner: https://example.com/v1/search?polygon[0][lat]=55.11&polygon[0][lon]=55.22
but at the moment fields with repeated custom messages are just ignored from generated http server and swagger spec.
Re #72
FieldMask is unsupported right now, generated code should hook into grpc-gateway FieldMask factories.
docs for the feature: https://grpc-ecosystem.github.io/grpc-gateway/docs/patch.html
I tried setting up a patch request gateway base from grpc_gateway documentation here.
Basically, I'm using field_mask and I'm trying to use the built-in behavior of grpc_gateway so that the client of the endpoint don't need to set the field_mask manually.
here is the callstack:
http: panic serving [::1]:58216: reflect: call of reflect.Value.Type on zero Value
goroutine 73 [running]:
net/http.(*conn).serve.func1(0xc001242000)
/usr/local/Cellar/go/1.13.4/libexec/src/net/http/server.go:1767 +0x139
panic(0x19ae0e0, 0xc000252d40)
/usr/local/Cellar/go/1.13.4/libexec/src/runtime/panic.go:679 +0x1b2
reflect.Value.Type(0x0, 0x0, 0x0, 0x1a30300, 0x400)
/usr/local/Cellar/go/1.13.4/libexec/src/reflect/value.go:1877 +0x166
github.com/golang/protobuf/jsonpb.(*Unmarshaler).unmarshalValue(0xc000252d00, 0x0, 0x0, 0x0, 0xc000039a60, 0x11, 0x20, 0x0, 0x100ebd8, 0x20)
/Users/user/Workspace/go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:720 +0x5d
github.com/golang/protobuf/jsonpb.(*Unmarshaler).UnmarshalNext(0xc000252d00, 0xc0011ac2c0, 0x1d54d00, 0x0, 0x0, 0x19a0101)
/Users/user/Workspace/go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:682 +0x166
github.com/grpc-ecosystem/grpc-gateway/runtime.decodeJSONPb(0xc0011ac2c0, 0x1acc1c0, 0x0, 0x1d6ef01, 0xc0011ac2c0)
/Users/user/Workspace/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:176 +0xa5
github.com/grpc-ecosystem/grpc-gateway/runtime.DecoderWrapper.Decode(0xc0011ac2c0, 0x1acc1c0, 0x0, 0x1d54d00, 0x0)
/Users/user/Workspace/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:149 +0x3f
github.com/utrack/clay/v2/transport/httpruntime.MarshalerPbJSON.Unmarshal(0x23e5380, 0x23e53c0, 0x23e5400, 0x23e34e0, 0x73024d8, 0xc000559fc0, 0x1acc1c0, 0x0, 0x1d59800, 0x73024d8)
/Users/user/Workspace/go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/mjson.go:59 +0x25f
github.com/org/project/pkg/transaction_service.glob..func11.1(0x1a6f620, 0xc0004100c0, 0xc00125b788, 0x100ebd8)
/Users/user/Workspace/go/src/github.com/org/project/pkg/transaction_service/transaction.pb.goclay.go:802 +0x168
github.com/org/project/pkg/transaction_service._TransactionService_UpdateTransaction_Handler(0x1a1c9a0, 0xc000520030, 0x1d5c640, 0xc00029f680, 0xc000276f20, 0xc000459590, 0x30, 0x30, 0x1a53f40, 0xc00125b8a0)
/Users/user/Workspace/go/src/github.com/org/project/pkg/transaction_service/transaction.pb.go:1956 +0x53
github.com/org/project/pkg/transaction_service.(*TransactionServiceDesc).RegisterHTTP.func7(0x1d57b40, 0xc000410080, 0xc0001c6500)
/Users/user/Workspace/go/src/github.com/org/project/pkg/transaction_service/transaction.pb.goclay.go:289 +0x102
net/http.HandlerFunc.ServeHTTP(...)
Any help would be appreciated. Thanks.
I want to allow delete request have a request body , but I don't konw how to setting ,
I use yaml to config my http define ,
--goclay_out: must not set request body when http method is DELETE except allow_delete_body option is true: DeleteUser
http:
rules:
- selector: user_center.ManageUserCenter.DeleteUser
delete: '/v1/example/userCenter/deleteUser'
body: 'data'
allow_delete_body: true
it does not work
Create new httpruntime.Marshaler that supports multipart messages and octet-stream
.
From this proto:
message ListCreativesRequest {
uint64 marketing_action_id = 1;
}
service CreativeService {
rpc ListCreative (ListCreativesRequest) returns (ListCreativeResponse) {
option (google.api.http) = {
get: "/v1/creatives"
};
}
}
will be generated wrong unmarshaller that doesn't fill ListCreativesRequest struct.
pattern_goclay_CreativeService_ListCreative_0 = "/v1/creatives"
unmarshaler_goclay_CreativeService_ListCreative_0 = func(r *http.Request, req *ListCreativesRequest) error {
return nil
}
Sample proto:
message UpdateCreativeRequest {
uint64 id = 1;
Creative creative = 2;
}
service CreativeService {
rpc UpdateCreative (UpdateCreativeRequest) returns (Creative) {
option (google.api.http) = {
put: "/v1/creatives/{id}"
body: "creative"
};
}
}
With the protoc-gen-go v1.20.0+ and new gRPC generator breaking compatibility (see grpc/grpc-go#3828), it seems that a big refactoring process will be needed (and frankly, current code calls for it).
This is a good chance to implement some things in a better way and throw away some others: for example, tricky output rules pre---module
flag.
This is an umbrella issue that will collect everything that will go to v3.0.0.
#33 might fix it since we've hit the limit on the current gateway registry, or wait until golang
and gogo
converge on a plan for their big refactoring.
It should be possible to crutch-fix that via reflection in the JSONMarshaler meanwhile.
Clay does not correct generate .pb.goclay.go file, if in proto-file exists into body exists one of proto-field with [(gogoproto.nullable) = false];
Example proto-file
rpc UserV1 (UserRequest) returns (UserResponse) {
option (google.api.http) = {
post: "UserV1"
body: "user"
};
}
...
message UserRequest {
User user = 1; [(gogoproto.nullable) = false];
}
Example piece generated code into .pb.goclay.go
...
req.User = &User{}
...
Error into generate template - https://github.com/utrack/clay/blob/master/cmd/protoc-gen-goclay/genhandler/template.go#L310
In github.com/utrack/clay/server/middlewares/mwhttp/closenotifier.go
github.com/go-chi/chi/middleware.CloseNotify is not available now. It's deleted from go-chi/chi
The package is deprecated since 1.4.0 (stable release 2020-04-16)
https://pkg.go.dev/github.com/golang/[email protected]/protoc-gen-go/generator?tab=doc
We need add Validator interface for Request struct. When request struct fetch data from request and if it implement Validator interface, we can validate data in request and return error before handler.
type Validator interface {
Valid() bool
}
I use prototool to call generation and got error:
$ prototool all "pb/"
could not interpret protoc line: E0615 15:20:09.519821 23865 handler.go:72] 114:18: expected ';', found '-': 0:
.
├── ...
├── pb
│ └── hello-world
│ └── hw.proto
└── ...
github.com/sirupsen/logrus v1.4.2
and
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
I browse swagger ui and input a ,b ,I get this error ,
oot@9de65925afcf:/workspaces/example# go run main.go
go: finding module for package github.com/utrack/clay/v2/transport/httptransport
go: finding module for package github.com/utrack/clay/v2/transport/httpclient
go: found github.com/utrack/clay/v2/transport/httpclient in github.com/utrack/clay/v2 v2.4.7
2020/06/05 16:26:12 http: panic serving 127.0.0.1:59764: reflect: call of reflect.Value.Type on zero Value
goroutine 51 [running]:
net/http.(*conn).serve.func1(0xc00017e000)
/usr/local/go/src/net/http/server.go:1772 +0x139
panic(0xa01be0, 0xc0003ea5e0)
/usr/local/go/src/runtime/panic.go:975 +0x3e3
reflect.Value.Type(0x0, 0x0, 0x0, 0x30, 0xa95320)
/usr/local/go/src/reflect/value.go:1872 +0x183
github.com/golang/protobuf/jsonpb.(*Unmarshaler).unmarshalValue(0xc0003ea560, 0x0, 0x0, 0x0, 0xc0003ca890, 0x10, 0x10, 0x0, 0x40e478, 0x20)
/go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:720 +0x5d
github.com/golang/protobuf/jsonpb.(*Unmarshaler).UnmarshalNext(0xc0003ea560, 0xc0000ec6e0, 0xca8260, 0x0, 0x0, 0xa08301)
/go/pkg/mod/github.com/golang/[email protected]/jsonpb/jsonpb.go:682 +0x166
github.com/grpc-ecosystem/grpc-gateway/runtime.decodeJSONPb(0xc0000ec6e0, 0xa7a880, 0x0, 0xcb4a01, 0xc0000ec6e0)
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:176 +0xa5
github.com/grpc-ecosystem/grpc-gateway/runtime.DecoderWrapper.Decode(0xc0000ec6e0, 0xa7a880, 0x0, 0xca8260, 0x0)
/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/runtime/marshal_jsonpb.go:149 +0x3f
github.com/utrack/clay/v2/transport/httpruntime.MarshalerPbJSON.Unmarshal(0x10db300, 0x10db340, 0x10db380, 0x10d9c00, 0x7f0c15b30d68, 0xc0000b9340, 0xa7a880, 0x0, 0xca9200, 0x7f0c15b30d68)
/go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/mjson.go:59 +0x25f
github.com/utrack/clay/doc/example/pb.glob..func2.1(0xa81180, 0xc0003b09f0, 0xc0001cb788, 0x40e478)
/workspaces/example/pb/sum.pb.goclay.go:234 +0x2a9
github.com/utrack/clay/doc/example/pb._Summator_Sum_Handler(0xa12880, 0x1106250, 0xcaa4a0, 0xc0003b09c0, 0xc0003a8440, 0xc0003b0060, 0x80, 0x80, 0xa950a0, 0xc000080000)
/workspaces/example/pb/sum.pb.go:209 +0x53
github.com/utrack/clay/doc/example/pb.(*SummatorDesc).RegisterHTTP.func1(0xca8360, 0xc0000b93c0, 0xc0000c0600)
/workspaces/example/pb/sum.pb.goclay.go:108 +0xe8
net/http.HandlerFunc.ServeHTTP(...)
/usr/local/go/src/net/http/server.go:2012
github.com/utrack/clay/v2/transport/httpruntime/httpmw.HeadersToGRPCMD.func1(0xca8360, 0xc0000b93c0, 0xc0000c0500)
/go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/httpmw/headers.go:45 +0x3d1
net/http.HandlerFunc.ServeHTTP(...)
/usr/local/go/src/net/http/server.go:2012
github.com/utrack/clay/v2/transport/httpruntime/httpmw.InjectTransportStream.func1(0xca8be0, 0xc0000b2380, 0xc0000c0400)
/go/pkg/mod/github.com/utrack/clay/[email protected]/transport/httpruntime/httpmw/headers.go:57 +0x23b
net/http.HandlerFunc.ServeHTTP(0xc0003a8140, 0xca8be0, 0xc0000b2380, 0xc0000c0400)
/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).routeHTTP(0xc0000c26c0, 0xca8be0, 0xc0000b2380, 0xc0000c0400)
/go/pkg/mod/github.com/go-chi/[email protected]+incompatible/mux.go:424 +0x278
net/http.HandlerFunc.ServeHTTP(0xc0003a80e0, 0xca8be0, 0xc0000b2380, 0xc0000c0400)
/usr/local/go/src/net/http/server.go:2012 +0x44
github.com/go-chi/chi.(*Mux).ServeHTTP(0xc0000c26c0, 0xca8be0, 0xc0000b2380, 0xc00017c400)
/go/pkg/mod/github.com/go-chi/[email protected]+incompatible/mux.go:81 +0x2b2
net/http.serverHandler.ServeHTTP(0xc0000b20e0, 0xca8be0, 0xc0000b2380, 0xc00017c400)
/usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc00017e000, 0xcaa3e0, 0xc000068000)
/usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2933 +0x35c
grpc client run correctly
You should store key "parameters" as well as "definitions"
Putting up this issue to ping everyone who watches this repo.
See the announcement here: ff09f40
Need a way to add my custom options to swagger.json.
I need this to add auth headers that I use.
We need to add an option to suppress non-fatal warnings to make clay work w/ prototool (see uber/prototool#128).
It will be great if I can paste some flag to generator which pass EmitDefaults value in default marshaler. This option is well discussed in that thread: grpc-ecosystem/grpc-gateway#233
I suppose to make it optional with false as default value not to change current behavior.
I'm talking about flag exactly, not to develop some custom httpruntime.Marshaler implementation and injection... If @utrack will be agree, I'll implement this feature.
There is a mistake with deferring GoPkg after getImplParam implementation (getImplTemplate refactoring) - no relative GoPkg got from getImplParam function
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.