golang / protobuf Goto Github PK
View Code? Open in Web Editor NEWGo support for Google's protocol buffers
License: BSD 3-Clause "New" or "Revised" License
Go support for Google's protocol buffers
License: BSD 3-Clause "New" or "Revised" License
I notice this tool have support generate grpc output, but I'd like to implement my own RPC for some reasons.
So I wonder if can you provide a tool generate rpc interfaces without dependencies on grpc, or can you give me some information about how to write my own generator?
One more question, to provide flexibility of my RPC framework, to easy change my generator, can I write a plugin use a python but generate code of golang?
Add this function to all_test.go
func TestMapFieldJsonMarshal(t *testing.T) {
m := &MessageWithMap{
NameMapping: map[int32]string{
1: "Rob",
4: "Ian",
8: "Dave",
},
}
_, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
}
It will fail with
json: unsupported type: map[int32]string
Is this work just not done yet or am I understanding something incorrectly or is this a bug?
I thought that json is the new text standard for proto3 and also that proto3 supports maps?
I would guess that go needs to support marshaling proto3 structures to json or am I interpreting this in the wrong way?
dhubler@duck ~/work/conf2/conf2-restconf $ go get -u github.com/golang/protobuf/proto
dhubler@duck ~/work/conf2/conf2-restconf $ go get -u github.com/golang/protobuf/protoc-gen-go
package github.com/golang/protobuf/protoc-gen-go
imports github.com/golang/protobuf/proto
imports github.com/golang/protobuf/protoc-gen-go/descriptor
imports github.com/golang/protobuf/protoc-gen-go/generator
imports github.com/golang/protobuf/protoc-gen-go/plugin
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc: use of internal package not allowed
dhubler@duck ~/work/conf2/conf2-restconf $ env | grep GO
GOROOT=/home/dhubler/work/conf2/go
GOPATH=/home/dhubler/work/conf2/conf2-restconf
dhubler@duck ~/work/conf2/conf2-restconf $ go version
go version devel +71859ef Mon Jun 22 06:57:19 2015 +0000 linux/amd64
Is there a reason for not having the README in rich text? Thought I'd ask before submitting a PR on the repo (not sure if its formatted this way intentionally).
The following program crashes with the panic:
package main
import (
pb "github.com/dvyukov/go-fuzz/examples/protobuf/pb"
"github.com/golang/protobuf/proto"
)
func main() {
data := []byte("\n\x02\n\x00")
v := new(pb.M24)
err := proto.Unmarshal(data, v)
if err != nil {
return
}
_, err = proto.Marshal(v)
if err != nil {
panic(err)
}
}
panic: proto: map has nil element
The proto used is:
message M24 {
map<string, M2> f = 1;
}
The message should either be successfully encoded or not decoded.
on commit 34a5f24
I have a second order plugin written in Go and after updating my golang/protobuf code and compiler locally, my plugin is no longer called.
I traced it down to a somewhat innocuous part of c22ae3c:
diff --git a/protoc-gen-go/generator/generator.go b/protoc-gen-go/generator/generator.go
index af0e15f..49f91d7 100644
--- a/protoc-gen-go/generator/generator.go
+++ b/protoc-gen-go/generator/generator.go
@@ -484,7 +484,7 @@ func (g *Generator) CommandLineParameters(parameter string) {
}
g.ImportMap = make(map[string]string)
- pluginList := "" // Default list of plugin names to enable (empty means all).
+ pluginList := "none" // Default list of plugin names to enable (empty means all).
for k, v := range g.Param {
switch k {
case "import_prefix":
Why the change from s/""/"none"/
? How (short of striking the above line) can I cause my second order plugin to continue to execute?
For example:
a.proto
package Test;
message A{
required string name=1;
}
b.proto
package Test;
import "a.proto";
message B{
required A a=1;
}
for this case ,protoc b.proto will see this :
import Test1 "a.pb"
thanks for reading my poor English
Add this to proto3.proto in proto/proto3_proto
message MessageWithMappy {
map<bool, bytes> byte_mapping = 3;
}
and run the Makefile in proto/proto3_proto
Then add this to size_test.go
{"map field with zero bytes", &proto3pb.MessageWithMappy{
ByteMapping: map[bool][]byte{
false: []byte{},
true: []byte{0x84, 0x78, 0x8e, 0x33, 0xcd, 0x12, 0x65, 0xa0, 0xd2, 0xca, 0xc4, 0xcd, 0x12, 0xe2, 0xcf, 0xf1, 0xbf, 0xfa, 0xf7, 0x5a, 0x3c, 0x6d, 0xdc, 0xcf, 0x9c, 0x92, 0x94, 0x3f, 0x4f, 0xb6, 0x31, 0x77, 0xe3, 0xda, 0x3f, 0x1b, 0x8e, 0x39, 0xe6, 0xa, 0x81, 0x61, 0xf0, 0xba, 0x74, 0x6},
},
}},
This gives the following failure
size_test.go:138: map field with zero bytes: Size(byte_mapping:<key:false value:"" > byte_mapping:<key:true value:"\204x\2163\315\022e\240\322\312\304\315\022\342\317\361\277\372\367Z<m\334\317\234\222\224?O\2661w\343\332?\033\2169\346\n\201a\360\272t\006" > ) = 56, want 7
size_test.go:139: map field with zero bytes: bytes: []byte{0x1a, 0x0, 0x0, 0x0, 0x0, 0x8, 0x0}
The error only happens with an empty byte slice.
This is probably a mistake in the way the map values try to reuse the setEncAndDec function with a combination of the proto3 syntax
case reflect.Uint8:
p.enc = (*Buffer).enc_slice_byte
p.dec = (*Buffer).dec_slice_byte
p.size = size_slice_byte
if p.proto3 {
p.enc = (*Buffer).enc_proto3_slice_byte
p.size = size_proto3_slice_byte
}
Maybe this should be
case reflect.Uint8:
p.enc = (*Buffer).enc_slice_byte
p.dec = (*Buffer).dec_slice_byte
p.size = size_slice_byte
if p.proto3 && notamapvalue {
p.enc = (*Buffer).enc_proto3_slice_byte
p.size = size_proto3_slice_byte
}
The following program crashes with a panic:
package main
import (
"github.com/golang/protobuf/proto"
"io/ioutil"
pb "protorepro/pb"
)
func main() {
proto.MarshalText(ioutil.Discard, &pb.M2{F: map[string]*pb.M1{"": nil}})
}
panic: reflect: call of reflect.Value.Interface on zero Value
goroutine 1 [running]:
runtime.gopanic(0x52e860, 0xc20800e4a0)
src/runtime/panic.go:461 +0x401 fp=0xc2080415a0 sp=0xc208041520
reflect.valueInterface(0x0, 0x0, 0x0, 0x542801, 0x0, 0x0)
src/reflect/value.go:906 +0x82 fp=0xc2080415f8 sp=0xc2080415a0
reflect.Value.Interface(0x0, 0x0, 0x0, 0x0, 0x0)
src/reflect/value.go:901 +0x4f fp=0xc208041630 sp=0xc2080415f8
github.com/golang/protobuf/proto.writeAny(0xc20800e400, 0x0, 0x0, 0x0, 0xc208070240, 0x0, 0x0)
src/github.com/golang/protobuf/proto/text.go:465 +0x632 fp=0xc2080417f8 sp=0xc208041630
github.com/golang/protobuf/proto.writeStruct(0xc20800e400, 0x536380, 0xc20800e3e0, 0xd9, 0x0, 0x0)
src/github.com/golang/protobuf/proto/text.go:295 +0xf89 fp=0xc208041da0 sp=0xc2080417f8
github.com/golang/protobuf/proto.marshalText(0x7f558bbca1e8, 0xc20800a5f0, 0x7f558bbca210, 0xc20800e3e0, 0x62c100, 0x0, 0x0)
src/github.com/golang/protobuf/proto/text.go:759 +0x440 fp=0xc208041ee0 sp=0xc208041da0
github.com/golang/protobuf/proto.MarshalText(0x7f558bbca1e8, 0xc20800a5f0, 0x7f558bbca210, 0xc20800e3e0, 0x0, 0x0)
src/github.com/golang/protobuf/proto/text.go:771 +0x57 fp=0xc208041f20 sp=0xc208041ee0
main.main()
protorepro.go:10 +0x129 fp=0xc208041f90 sp=0xc208041f20
The proto used is:
syntax = "proto2";
package example;
message M1 {
optional uint32 f = 1;
}
message M2 {
map<string, M1> f = 1;
}
go test github.com/golang/protobuf/proto
# github.com/golang/protobuf/proto
proto3_test.go:38:2: no buildable Go source files in C:\go\gopkg.stable\src\github.com\golang\protobuf\proto\proto3_proto
FAIL github.com/golang/protobuf/proto [setup failed]
The first letter of type/field name is always uppercase in generated pb.go codes, making it no way to create non-exported struct/field with protobuf in Go. Is this by design?
The README file mentions to create a Makefile like this:
include $(GOROOT)/src/Make.$(GOARCH)
...
include $(GOROOT)/src/Make.pkg
But those files no (longer?) exists in go 1.4
What's the content of those files?
I am unable to build a package when a proto file is included from a separate package.
src/foo/foo.proto
package foo;
message Foo
{
optional string foo = 1;
}
protoc --go_out=src/ --proto_path=src/ src/foo/foo.proto
src/foo/foo.pb.go
// Code generated by protoc-gen-go.
// source: foo/foo.proto
// DO NOT EDIT!
/*
Package foo is a generated protocol buffer package.
It is generated from these files:
foo/foo.proto
It has these top-level messages:
Foo
*/
package foo
import proto "github.com/golang/protobuf/proto"
import math "math"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf
type Foo struct {
Foo *string `protobuf:"bytes,1,opt,name=foo" json:"foo,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Foo) Reset() { *m = Foo{} }
func (m *Foo) String() string { return proto.CompactTextString(m) }
func (*Foo) ProtoMessage() {}
func (m *Foo) GetFoo() string {
if m != nil && m.Foo != nil {
return *m.Foo
}
return ""
}
func init() {
}
src/bar/bar.proto
package bar;
import "foo/foo.proto";
message Bar
{
optional foo.Foo foo = 1;
optional string bar = 2;
}
protoc --go_out=src/ --proto_path=src/ src/bar/bar.proto
src/bar/bar.pb.go
// Code generated by protoc-gen-go.
// source: bar/bar.proto
// DO NOT EDIT!
/*
Package bar is a generated protocol buffer package.
It is generated from these files:
bar/bar.proto
It has these top-level messages:
Bar
*/
package bar
import proto "github.com/golang/protobuf/proto"
import math "math"
import foo "foo/foo.pb"
// Reference imports to suppress errors if they are not otherwise used.
var _ = proto.Marshal
var _ = math.Inf
type Bar struct {
Foo *foo.Foo `protobuf:"bytes,1,opt,name=foo" json:"foo,omitempty"`
Bar *string `protobuf:"bytes,2,opt,name=bar" json:"bar,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
func (m *Bar) Reset() { *m = Bar{} }
func (m *Bar) String() string { return proto.CompactTextString(m) }
func (*Bar) ProtoMessage() {}
func (m *Bar) GetFoo() *foo.Foo {
if m != nil {
return m.Foo
}
return nil
}
func (m *Bar) GetBar() string {
if m != nil && m.Bar != nil {
return *m.Bar
}
return ""
}
func init() {
}
go build foo
go build bar
src/bar/bar.pb.go:18:8: cannot find package "foo/foo.pb" in any of:
/usr/local/Cellar/go/1.3/libexec/src/pkg/foo/foo.pb (from $GOROOT)
/Users/patrick/Code/workspace/src/foo/foo.pb (from $GOPATH)
When I change import foo "foo/foo.pb" to import foo "foo" the package will build correctly. Is this an issue with my use of the generated files or an issue with code generation? Thanks.
| have installed the plugin in my $GOPATH in order to give it a go, and I have created a .proto file with proto3 syntax where I got rid of all the required keywords and started the file with ' syntax="proto3" ', however I keep getting this compilation error message:
" Unrecognized syntax identifier "proto3". This parser only recognizes "proto2"."
P.S: when I change from proto3 to proto2 the compilation succeeded however I have of course to invoke the required keywords again because this feature is not supported in proto2.
The README suggests using "go get -u" to install protobuf. However, it doesn't work.
$ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
package github.com/golang/protobuf/proto: /Users/paul/Code/go/src/github.com/golang/protobuf is from [email protected]:golang/protobuf, should be from https://github.com/golang/protobuf
package github.com/golang/protobuf/protoc-gen-go: /Users/paul/Code/go/src/github.com/golang/protobuf is from [email protected]:golang/protobuf, should be from https://github.com/golang/protobuf
$ go version
go version go1.5beta1 darwin/amd64
$ go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
package github.com/golang/protobuf/proto
imports github.com/golang/protobuf/protoc-gen-go
imports github.com/golang/protobuf/protoc-gen-go/descriptor
imports github.com/golang/protobuf/protoc-gen-go/generator
imports github.com/golang/protobuf/protoc-gen-go/plugin
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc
imports github.com/golang/protobuf/protoc-gen-go/internal/grpc: use of internal package not allowed
Empty map is cloned as nil via proto.Clone. nil and empty map are different in Go.
Is it possible to define a message
with custom tags? For example defining a json name that isn't just the lowercase field name or adding a bson tag.
The following program crashes with the following message:
package main
import (
"bytes"
"fmt"
pb "github.com/dvyukov/go-fuzz/examples/protobuf/pb"
"github.com/golang/protobuf/proto"
)
func main() {
data := []byte("\n\x010")
v := new(pb.M25)
err := proto.Unmarshal(data, v)
if err != nil {
panic(err)
}
var buf bytes.Buffer
err = proto.MarshalText(&buf, v)
if err != nil {
panic(err)
}
v2 := new(pb.M25)
err = proto.UnmarshalText(buf.String(), v2)
if err != nil {
fmt.Printf("failed to UnmarshalText: %q\n", buf.Bytes())
panic(err)
}
}
failed to UnmarshalText: "/* 3 unknown bytes */\n1: \"0\"\n"
panic: line 1.0: unexpected byte 0x2f
MarshalText should not produce text that is unparsable by UnmarshalText.
Either MarshalText should write unknown bytes differently, or UnmarshalText accept the current output of MarshalText.
on commit 34a5f24
From http://golang.org/doc/effective_go.html#slices
Slices hold references to an underlying array,
and if you assign one slice to another,
both refer to the same array.
When I generate the following proto holding the following messages
message AmessageWithAString
{
required string aString = 1;
}
message RepeatedString
{
repeated AmessageWithAString PointerToSlice = 1;
}
the generator generates
type RepeatedString struct {
PointerToSlice []*AmessageWithAString `protobuf:"bytes,1,rep" json:"PointerToSlice,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
shouldn't it generate
PointerToSlice []AmessageWithAString
instead?
Hi,
what is the plan for adding support for google/protobuf/timestamp.proto, e.g. generating Time fields in the proto structs?
Thanks
I have proto file
message MsgErrCode {
required int32 err_code = 1;
optional string reason = 2;
}
After generated, I got go file
type MsgErrCode struct {
ErrCode *int32 `....`
Reason *string `....`
}
Notice that the field generated is pointer, It make me hard to use the struct,
I can't do &MsgErrCode{ErrCode: int32(12), Reason: "hello world"}
I have to do &MsgErrCode{ErrCode: proto.Int32(12), Reason: proto.String("hello world")}
Passing multiple packages' worth of *.proto files currently fails when using (protoc-gen-go)[https://github.com/golang/protobuf/blob/master/protoc-gen-go/generator/generator.go#L612:L614].
The same functionality is supported by other generators (at least the C++ generator shipped with protoc). protoc-gen-go should support this as well.
Hi, when generating go protos like
protoc --go_out=plugins=grpc:. /.proto
it fails with error:inconsistent package names: blabla
I guess the assumptions is that it will be called only for files in the same directory. Are there plans to fix that?
I would be happy to do the work and commit it over, but before doing any work on this, wanted to see how open the maintainers were to it. Thoughts?
Also addresses #49
In the proto3 there are encodings for different types, in particular the Timestamp type. Where is that code implemented? I am having a hard time figuring it out. https://developers.google.com/protocol-buffers/docs/proto3#json
cc @sym3tri
go version devel +87054c4 Wed Apr 22 02:50:48 2015 +0000 linux/amd64
golang/protobuf is on 655cdfa
protoc is on fe7b566
Protobuf is:
syntax = "proto2";
package main;
message M {
map<int32, int32> f = 1;
}
Program is:
package main
import (
"github.com/golang/protobuf/proto"
)
func main() {
m := &M{}
data := []byte("\v\x00")
proto.Unmarshal(data, m)
}
Crashes with:
panic: reflect: call of reflect.Value.SetMapIndex on zero Value
goroutine 1 [running]:
runtime.gopanic(0x530860, 0xc2080104e0)
reflect.flag.mustBeExported(0x0)
reflect.Value.SetMapIndex(0x5058e0, 0xc208010440, 0xd5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
github.com/golang/protobuf/proto.(*Buffer).dec_new_map(0xc208052f70, 0xc20806c480, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.(*Buffer).unmarshalType(0xc208052f70, 0x7f70a263e198, 0x5384e0, 0xc208048070, 0xc208010400, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.(*Buffer).Unmarshal(0xc208052f70, 0x7f70a263e2d8, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.UnmarshalMerge(0xc20800a570, 0x2, 0x8, 0x7f70a263e2d8, 0xc208010440, 0x0, 0x0)
github.com/golang/protobuf/proto.Unmarshal(0xc20800a570, 0x2, 0x8, 0x7f70a263e2d8, 0xc208010440, 0x0, 0x0)
main.main()
Take this generated code:
type FlagId struct {
Deployment *DeploymentId `protobuf:"bytes,1,opt,name=deployment" json:"deployment,omitempty"`
Name string `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"`
}
func (m *FlagId) Reset() { *m = FlagId{} }
func (m *FlagId) String() string { return proto.CompactTextString(m) }
func (*FlagId) ProtoMessage() {}
func (m *FlagId) GetDeployment() *DeploymentId {
if m != nil {
return m.Deployment
}
return nil
}
What is the purpose of this Getter? It will not be called on a nil
m *FlagId
.
I actually hoped it would create an new instance of the type behind the Deployment
field, but it seems to not do anything other than a normal m.Deployment
would.
I really hoped it would create a new instance if it was nil, this way I could solve the problem in without reflection:
grpc-ecosystem/grpc-gateway#32
As almost all other tools (e.g. oauth2) uses it, why does this repo use the barebone github path instead of the vanity ones? Wouldn't it make more sense to switch over?
When I import some of google's proto definitions the generated code imports a package that does not exist.
Example:
syntax = "proto3";
import "google/protobuf/timestamp.proto";
package bar;
message Foo {
google.protobuf.Timestamp start = 1;
google.protobuf.Timestamp finish = 2;
}
Generated code:
package bar
import proto "github.com/golang/protobuf/proto"
import google_protobuf "google/protobuf"
...
The import of google_protobuf
fails.
google_protobuf
package and adding it manually to my GOPATH
?Would it be possible to add this function to proto/encode.go ?
func NewRequiredNotSetError(field string) *RequiredNotSetError {
return &RequiredNotSetError{field}
}
That way generated code can also return a RequiredNotSetError with a field string.
Currently I have to:
return new(RequiredNotSetError)
Which is cool since then users can still type check the error, but it is pretty useless for debugging purposes.
I know that unknown fields have been removed from proto3, but I am trying to get an explanation about why this change was made and if there is any way to replicate that behavior in proto3.
Thanks so much.
In proto2 fields were generated as pointers, which allowed us to check if a field was specified or not. This no longer seems to be the case. For example:
proto2:
syntax = "proto2";
message Foo {
required int32 ID = 1;
optional int32 SortOrder = 2;
}
gives
type Foo struct {
ID *int32 `protobuf:"varint,1,req" json:"ID,omitempty"`
SortOrder *int32 `protobuf:"varint,2,opt" json:"SortOrder,omitempty"`
XXX_unrecognized []byte `json:"-"`
}
proto3:
syntax = "proto3";
message Foo {
int32 ID = 1;
optional int32 SortOrder = 2;
}
gives
type Foo struct {
ID int32 `protobuf:"varint,1,opt" json:"ID,omitempty"`
SortOrder int32 `protobuf:"varint,2,opt" json:"SortOrder,omitempty"`
}
Is this the expected behavior, and if so how are we supposed to differentiate a client making a call with SortOrder = 0
vs one making a call and not specifying the SortOrder
?
I use -I$GOPATH/src
to include protos, but it's path does not match somtimes.
As golang does not work in simillar way with other langauges, an extra option is required.
Example usecase:
golang package: appcore-project/cdt-go/country
protobuf: appcore-project/protos/appcore/cdt/country/country.proto
I have a Protobuf message that takes an optional message of a type I created. However, the optional message type is just a wrapper of nested types. The goal was to be able to specify a thing generically by saying that it's simply something of a Type and not a RedType or GreenType. Polymorphism.
I've messed with all of this to make it concise and easier to read.
package interfaces;
message Wrapper {
optional Type data = 1;
}
message Type {
message RedType {
required int64 id;
required string red_type_property;
}
message GreenType {
required int64 id;
required string green_type_property;
}
}
The issue is, when I go to work with this in Go:
mydata := &interfaces.Type_RedType{
Id: 20,
RedTypeProperty: "propval",
}
mywrapper := &interfaces.Wrapper{
Data: mydata,
}
Not surprisingly, you can't use something that is a Type_RedType
in place of a more general Type
when setting the struct value of Data
.
cannot use mydata (type *interfaces.Type_RedType) as type *interfaces.Type in field value
Is there a language feature of Go I can use here? I tried a type conversion from RedType
to Type
without expecting much and it didn't work. Am I misusing this library? Or is this a terrible idea to begin with, which is what has landed me here?
Any & all help appreciated. Hopefully this is an easy one to answer.
I'm trying to deploy an application on GAE VM. Although I'm not using the protobuf package in my app the goapp command throws the error below:
/Users/xyz/google-cloud-sdk/bin/goapp deploy ./
github.com/golang/protobuf/proto/decode.go:219: undefined: structPointer
github.com/golang/protobuf/proto/decode.go:219: undefined: field
github.com/golang/protobuf/proto/properties.go:65: undefined: structPointer
github.com/golang/protobuf/proto/properties.go:73: undefined: structPointer
github.com/golang/protobuf/proto/properties.go:82: undefined: structPointer
github.com/golang/protobuf/proto/properties.go:166: undefined: field
The following program fails:
package main
import (
"bytes"
"fmt"
"reflect"
pb "github.com/dvyukov/go-fuzz/examples/protobuf/pb"
"github.com/golang/protobuf/proto"
)
func main() {
data := []byte("\xf5\a\xb3\xc4\xe1\xbf")
v := new(pb.M18)
err := proto.Unmarshal(data, v)
if err != nil {
panic(err)
}
var buf bytes.Buffer
err = proto.MarshalText(&buf, v)
if err != nil {
fmt.Printf("failed to MarshalText: %#v\n", v)
panic(err)
}
v2 := new(pb.M18)
err = proto.UnmarshalText(buf.String(), v2)
if err != nil {
fmt.Printf("failed to UnmarshalText: %q\n", buf.Bytes())
panic(err)
}
if !reflect.DeepEqual(v, v2) {
fmt.Printf("v0: %#v\n", v)
fmt.Printf("v2: %#v\n", v2)
panic(fmt.Sprintf("non idempotent text marshal of %T", v))
}
}
proto: failed getting extension: unexpected EOF
v0: &example.M18{F0:(*string)(nil), XXX_extensions:map[int32]proto.Extension{126:proto.Extension{desc:(*proto.ExtensionDesc)(nil), value:interface {}(nil), enc:[]uint8{0xf5, 0x7, 0xb3, 0xc4, 0xe1, 0xbf}}}, XXX_unrecognized:[]uint8(nil)}
v2: &example.M18{F0:(*string)(nil), XXX_extensions:map[int32]proto.Extension(nil), XXX_unrecognized:[]uint8(nil)}
panic: non idempotent text marshal of *example.M18
There are 2 issues here:
Definition of the protobuf is:
message M18 {
optional string f0 = 1;
extensions 100 to 199;
}
extend M18 {
optional int32 f1 = 126;
}
on commit 34a5f24
The current implementation doesn't allow key-based indexing or the use of range
. It appears that the current solution is to create a repeated K/V message. Which isn't really the same as a map.
when I run protoc on Mac's terminal, will stop on
data, err := ioutil.ReadAll(os.Stdin)
Add this to MessageWithMap in proto/testdata/test.proto
map<string, string> stringy = 4;
Add this to size_test.go in proto
{"map with string and string", &pb.MessageWithMap{
Stringy: map[string]string{"YGeGc18B1eey5ifSdWMn2c21wRAgY90YvH4HmCOeSa9q2pfAIJa9WiGOOZGZKxZgoJo7VXnQumvsch": "RggKn1pmtK98gnMhD2J0GHy9R4IFPZQwjFeyphRQzEqXkVn"},
}},
You should get this error:
size_test.go:141: map with string and string: Size(stringy:<key:"YGeGc18B1eey5ifSdWMn2c21wRAgY90YvH4HmCOeSa9q2pfAIJa9WiGOOZGZKxZgoJo7VXnQumvsch" value:"RggKn1pmtK98gnMhD2J0GHy9R4IFPZQwjFeyphRQzEqXkVn" > ) = 131, want 132
size_test.go:142: map with string and string: bytes: []byte{0x22, 0x81, 0x1, 0xa, 0x4e, 0x59, 0x47, 0x65, 0x47, 0x63, 0x31, 0x38, 0x42, 0x31, 0x65, 0x65, 0x79, 0x35, 0x69, 0x66, 0x53, 0x64, 0x57, 0x4d, 0x6e, 0x32, 0x63, 0x32, 0x31, 0x77, 0x52, 0x41, 0x67, 0x59, 0x39, 0x30, 0x59, 0x76, 0x48, 0x34, 0x48, 0x6d, 0x43, 0x4f, 0x65, 0x53, 0x61, 0x39, 0x71, 0x32, 0x70, 0x66, 0x41, 0x49, 0x4a, 0x61, 0x39, 0x57, 0x69, 0x47, 0x4f, 0x4f, 0x5a, 0x47, 0x5a, 0x4b, 0x78, 0x5a, 0x67, 0x6f, 0x4a, 0x6f, 0x37, 0x56, 0x58, 0x6e, 0x51, 0x75, 0x6d, 0x76, 0x73, 0x63, 0x68, 0x12, 0x2f, 0x52, 0x67, 0x67, 0x4b, 0x6e, 0x31, 0x70, 0x6d, 0x74, 0x4b, 0x39, 0x38, 0x67, 0x6e, 0x4d, 0x68, 0x44, 0x32, 0x4a, 0x30, 0x47, 0x48, 0x79, 0x39, 0x52, 0x34, 0x49, 0x46, 0x50, 0x5a, 0x51, 0x77, 0x6a, 0x46, 0x65, 0x79, 0x70, 0x68, 0x52, 0x51, 0x7a, 0x45, 0x71, 0x58, 0x6b, 0x56, 0x6e}
Aka: https://developers.google.com/protocol-buffers/docs/proto3#oneof
Specifically: protos including oneof do produce working .pb.proto files, but there doesn't seem to be any of the useful bits of Oneof implemented (selecting on which field is set, setting a field resetting the rest).
I'm sure this is being tracked internally somewhere, but I thought it would be useful to have an external issue for it for people to watch.
https://github.com/golang/protobuf/blob/master/protoc-gen-go/internal/grpc/grpc.go#L116
Depending on the plugin order generation this could result in variables being declared in between import statements, which is invalid go.
Luckily golang/protobuf has just one plugin, so this should not be a problem, but in go1.5 this is a problem for gogo/protobuf.
gogo/protobuf#83
I could easily fix this in gogo/protobuf, but I think this might cause problems for you later as well. At which point I'll have to undo my fix and merge in your fix :(
Is it an aim of protoc-gen-go
to produce names which pass golint? In particular, I find it is not capitalizing abbreviations as I expect (Id vs ID).
A contrived example is
message Args {
int64 id = 1;
string api_endpoint = 2;
}
which produces
type Args struct {
Id int64 `protobuf:"varint,1,opt,name=id" json:"id,omitempty"`
ApiEndpoint string `protobuf:"bytes,2,opt,name=api_endpoint" json:"api_endpoint,omitempty"`
}
I prefer the names ID
and APIEndpoint
which pass golint. Currently, I must wrap these public fields in getters and be sure to use them to avoid propagating use of the generated names throughout my codebase.
For this project, is passing golint a goal, a loose aspiration, or no stance?
Thanks a bunch for your work on this :)
Related:
golang/lint#89
Using the jsonpb package, I can marshal a proto3 message to JSON with enums as strings.
However, I cannot go in the reverse direction. The Unmarshal function returns an error when I try to unmarshal JSON with string enums.
I expected this to work how it does with proto2 messages.
Failing round-trip example here:
package main
import (
"github.com/golang/protobuf/jsonpb"
pb "github.com/golang/protobuf/proto/proto3_proto"
)
func main() {
msg := &pb.Message{Hilarity: pb.Message_SLAPSTICK}
marshaller := jsonpb.Marshaller{EnumsAsString: true}
data, _ := marshaller.MarshalToString(msg)
if err := jsonpb.UnmarshalString(data, msg); err != nil {
panic(err)
}
}
This code panics with json: cannot unmarshal string into Go value of type proto3_proto.Message_Humour
.
This seems confusing because the %+v
verb would typically include these fields when used with a generic struct.
Is there some additional reason that our Stringer chooses to elide these fields?
This is a request to support generating code that uses lowercase struct names so they are not exported in the package they target. The motivation is to reduce the API footprint exposed to end users. For example, in a project I am working on I define a set of methods that use the protobuf structs internally, but I don't want them to be exposed themselves: http://godoc.org/github.com/chop-dbhi/origins/dal
When defining a service in a proto file, the result of the compilation generates a MyNameServer instead of MyNameService. I've also opened the issue in the grpc-go repo because I don't know where this should be taken into account (grpc/grpc-go#114) I'll cose whichever is not in the proper place.
When I run go get -u github.com/golang/protobuf/{proto,protoc-gen-go}
as instructed in README I keep getting the following error (with Go 1.4.2 on Linux and Mac):
$ go get -u -t -v -f github.com/golang/protobuf/{proto,protoc-gen-go}
github.com/golang/protobuf (download)
package github.com/golang/protobuf/proto
imports github.com/golang/protobuf/proto/proto3_proto
imports github.com/golang/protobuf/proto/proto3_proto
imports github.com/golang/protobuf/proto/proto3_proto: no buildable Go source files in /home/dmitris/go/src/github.com/golang/protobuf/proto/proto3_proto
Should this error be ignored? (the protobuf directory does get checked out, and you can go install ./...
in it). But ignoring errors seem contrary to the Go spirit ๐ - maybe it would be possible to add a dummy file in https://github.com/golang/protobuf/tree/master/proto/proto3_proto to prevent the error (not sure what is the standard and recommended way to deal with such - would be interested since it comes up in other places as well). Thanks!
Is there an idiomatic way to support Any's in messages in Go? I have been struggling with reflect for a couple of days, and I think if I'm having this much trouble, I must be doing it wrong.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.