ameyag / protobuf-java-format Goto Github PK

When field like (which was generated using XMLFomat.printToString):

<email>[email protected]</email>

is parsed, tokenizer detects these 2 tokens:
'juraj@michalak' and 'com'
The error output is:
Caused by: com.googlecode.protobuf.format.XmlFormat$ParseException: 1:71: 
Expected ">".
        at com.googlecode.protobuf.format.XmlFormat$Tokenizer.parseException(XmlFormat.java:668)
        at com.googlecode.protobuf.format.XmlFormat$Tokenizer.consume(XmlFormat.java:467)
        at com.googlecode.protobuf.format.XmlFormat.consumeClosingElement(XmlFormat.java:791)
        at com.googlecode.protobuf.format.XmlFormat.mergeField(XmlFormat.java:875)
        at com.googlecode.protobuf.format.XmlFormat.handleObject(XmlFormat.java:993)
        at com.googlecode.protobuf.format.XmlFormat.handleValue(XmlFormat.java:886)
        at com.googlecode.protobuf.format.XmlFormat.mergeField(XmlFormat.java:866)
        at com.googlecode.protobuf.format.XmlFormat.merge(XmlFormat.java:774)
        at com.googlecode.protobuf.format.XmlFormat.merge(XmlFormat.java:722)
        at com.protobufexample.ListPeople.main(ListPeople.java:64)

Correct would be only one token:
'[email protected]'

protobuf-java-format 1.2

What steps will reproduce the problem?
1.create a simple protobuf message, message CharTest{required string a=1}
2.create a protobuf instance, with a=
3. CharTest m CharTest.newBuilder().setA("my.test").build();
   String xmlText =  XmlFormat.printToString(m);
   UserAuthReq.Builder builder2 = UserAuthReq.newBuilder();   
   XmlFormat.merge(xmlText, builder2);

What is the expected output? What do you see instead?
It should run without exception. 
Actual output:
com.google.protobuf.XmlFormat$ParseException: 1:33: Expected ">".
        at com.google.protobuf.XmlFormat$Tokenizer.parseException(XmlFormat.java:662)
        at com.google.protobuf.XmlFormat$Tokenizer.consume(XmlFormat.java:461)
        at com.google.protobuf.XmlFormat.consumeClosingElement(XmlFormat.java:785)
        at com.google.protobuf.XmlFormat.mergeField(XmlFormat.java:869)
        at com.google.protobuf.XmlFormat.merge(XmlFormat.java:768)
        at com.google.protobuf.XmlFormat.merge(XmlFormat.java:734)
        at com.google.protobuf.XmlFormat.merge(XmlFormat.java:709)


What version of the product are you using? On what operating system?
protobuf-java-format 1.1, protobuf-2.3, Linux

This seems to be the way how to handle the nextToken. it doens't match the 
Token pattern.

I see that you have taken away all of my permissions to be able to contribute 
to the project without providing any notice or explanation.  Could you please 
fix the permissions so that I could continue to contribute!

Thanks,
Alex

BJF could be fixed to allow JSON field names in camelCase or sname_case. 

What steps will reproduce the problem?

1. Make a .proto file with a snake_case field_name.
2. Generate JSON from protobuf using https://github.com/chrisdew/protobuf/
3. The JSON will have a camelCase fieldName
4. PJF will fail to parse the JSON.

e.g.

1. download pjf_empty_object.tgz
2. extract it (tar -xzvf pjf_empty_object.tgz)
3. cd pjf_empty_object
4. make clean && make && make run


What is the expected output? What do you see instead?

expected:

textJson: {"text":"foo"}
textPdu: {"text": "foo"}
authAckJson: {"authAck":{}}
authAckPdu: {"authAck": {}}

actual:

textJson: {"text":"foo"}
textPdu: {"text": "foo"}
authAckJson: {"authAck":{}}
com.googlecode.protobuf.format.JsonFormat$ParseException: 1:13: Expected 
identifier. -}
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.parseException(JsonFormat.java:765)
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.consumeIdentifier(JsonFormat.java:575)
    at com.googlecode.protobuf.format.JsonFormat.handleMissingField(JsonFormat.java:970)
    at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:936)
    at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:870)
    at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:818)
    at com.chrisdew.Main.main(Main.java:26)
authAckPdu: {}


What version of the product are you using? On what operating system?

$ ls libs/protobuf*
libs/protobuf-2.5.0.jar  libs/protobuf-java-format-1.2.jar

$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

$ uname -a
Linux chris-work 3.2.0-45-generic #70-Ubuntu SMP Wed May 29 20:12:06 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux


Please provide any additional information below.

pjfeoi.proto:

package com.chrisdew.pjfeoi;

message Pdu {
  optional string  text     = 4;
  optional AuthAck auth_ack = 5;
}
message AuthAck {
}


Main.java:

package com.chrisdew;

import com.google.protobuf.UninitializedMessageException;
import com.googlecode.protobuf.format.JsonFormat;
import com.chrisdew.pjfeoi.Pjfeoi;

public class Main {
        public static void main(String[] args) {
                String textJson = "{\"text\":\"foo\"}";
                System.out.println("textJson: " + textJson);

                Pjfeoi.Pdu.Builder builderForText = Pjfeoi.Pdu.newBuilder();
                try {
                        JsonFormat.merge(textJson, builderForText);
                } catch (JsonFormat.ParseException e) {
                        e.printStackTrace();
                }
                Pjfeoi.Pdu textPdu = builderForText.build();
                System.out.println("textPdu: " + JsonFormat.printToString(textPdu));

                String authAckJson = "{\"authAck\":{}}";
                System.out.println("authAckJson: " + authAckJson);

                Pjfeoi.Pdu.Builder builderForAuthAck = Pjfeoi.Pdu.newBuilder();
                try {
                        JsonFormat.merge(authAckJson, builderForAuthAck);
                } catch (JsonFormat.ParseException e) {
                        e.printStackTrace();
                }
                Pjfeoi.Pdu authAckPdu = builderForAuthAck.build();
                System.out.println("authAckPdu: " + JsonFormat.printToString(authAckPdu));
        }
}

P.S. this was previously 
https://code.google.com/p/protobuf-java-format/issues/detail?id=48

What steps will reproduce the problem?
1.Use JavaPropsFormat.printToString(myAddressBookProtoObj) to get a properties 
string
2.use this string to generate a proto object

What is the expected output? What do you see instead?
addressBookObject should be like this: 
addressbook:{
person {
  name: "jack"
  id: 1
  email: "[email protected]"
  phone {
    number: "1234"
    type: HOME
  }
  phone {
    number: "5678"
    type: WORK
  }
}
}
However, there is an exception occured:
Exception in thread "main" java.lang.NullPointerException
    at com.google.protobuf.GeneratedMessage$FieldAccessorTable.access$100(Unknown Source)
    at com.google.protobuf.GeneratedMessage$Builder.newBuilderForField(Unknown Source)
    at com.googlecode.protobuf.format.JavaPropsFormat.mergeField(JavaPropsFormat.java:937)
    at com.googlecode.protobuf.format.JavaPropsFormat.merge(JavaPropsFormat.java:850)
    at com.googlecode.protobuf.format.JavaPropsFormat.merge(JavaPropsFormat.java:795)
    at com.example.tutorial.ProtoTest.print(ProtoTest.java:160)
    at com.example.tutorial.ProtoTest.main(ProtoTest.java:19)


What version of the product are you using? On what operating system?
1.2

Please provide any additional information below.
There is same issue for XmlFormat and JsonFormat.
It seems that JavaPropsFormat can printToString successfully and the properties 
string is as follows:
person[0].name="jack"
person[0].id=1
person[0].email="[email protected]"
person[0].phone[0].number="1234"
person[0].phone[0].type=HOME
person[0].phone[1].number="5678"
person[0].phone[1].type=WORK

What steps will reproduce the problem?
simple code example

import com.google.protobuf.ByteString;
import com.google.protobuf.Message;
import com.googlecode.protobuf.format.JsonFormat;
import com.viaden.network.yoga.core.api.YogaCoreDomain;

/**
 * protobuf message
 * <p/>
 * {@code message TestObject { required bytes data = 1; }}
 */
public class JacksonTest {

    private static final char BELL_CHARACTER = '\u0007';

    public static void main(String[] args) throws Exception {
        final YogaCoreDomain.TestObject.Builder testObject = YogaCoreDomain.TestObject.newBuilder();
        testObject.setData(ByteString.copyFrom(new byte[]{BELL_CHARACTER}));

        final Message.Builder builder = YogaCoreDomain.TestObject.getDefaultInstance().newBuilderForType();
        final String result = JsonFormat.printToString(builder.mergeFrom(testObject.build()).build());
    }
}

What is the expected output? What do you see instead?
expected:
{"data":"\u0007"}
actual
{"data":"\a"}

What version of the product are you using? On what operating system?
1.2

Please provide any additional information below.

XmlFormat requires a unit test

To reporoduce convert to XML data containing special symbols e.g. '<' or '&'

Expected:  @amp; @lt; etc.
Got: &, <

To fix: add precessing for special symbols in  method
  XMLFormat.escapeBytes(ByteString input)

What steps will reproduce the problem?
1. prepare this json {"name":"Aamore</1"}
2. merge with JsonFormat.merge(json, aProtoObj);
3.com.googlecode.protobuf.format.JsonFormat$ParseException: 1:9: Invalid escape 
sequence: '\/'
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.parseException(JsonFormat.java:765)

What version of the product are you using? On what operating system?
protobuf-java-format 1.2 on mac osx

slash can be optionally escaped, here is not handled

JsonFormat require a unit test

What steps will reproduce the problem?
1. Try to convert a <object><string></string></object> into java instance,
will get a object.string equal to "</" instant of expected ""

Fix:
--- XmlFormat.java  2010-06-03 17:06:18.000000000 +0100
+++ XmlFormat.java  2010-10-14 17:52:51.500656265 +0100
@@ -392,6 +394,14 @@
             return currentToken.length() == 0;
         }

+        public String getCurrentToken() {
+            return currentToken;
+        }
+
+        public void setCurrentToken(String string) {
+            currentToken = string;
+        }
+
         /**
          * Advance to the next token.
          */
@@ -857,6 +867,10 @@

         tokenizer.consume(">");

+        if(tokenizer.getCurrentToken().equals("</")) {
+            tokenizer.setCurrentToken("");
+        }
+
         Object value = handleValue(tokenizer, extensionRegistry, builder, field, extension);

         if (field.isRepeated()) {



PS: Also should allow for short tags e.g. <data/> not only <data></data> which 
will be possible with this fix.

Problem:
JsonFormat Line ~1281:

case 'u':
// UTF8 escape
code = 
(16 * 3 * digitValue(input.charAt(i+1))) +
(16 * 2 * digitValue(input.charAt(i+2))) +
(16 * digitValue(input.charAt(i+3))) +
digitValue(input.charAt(i+4));
i = i+4;
result[pos++] = (byte) code;
break;

Correct (not multiplied but by the Power of 16):
case 'u':
// UTF8 escape
code = 
(16 * 16 * 16 * digitValue(input.charAt(i+1))) +
(16 * 16 * digitValue(input.charAt(i+2))) +
(16 * digitValue(input.charAt(i+3))) +
digitValue(input.charAt(i+4));
i = i+4;
result[pos++] = (byte) code;
break;

I attached a version with handels the Umlaute correct.

What steps will reproduce the problem?
1. Build a Message containing a String with Umlaute (äöüÄÖÜäàéè)
2. JsonFormat.print(req.build(), sb);
3. JsonFormat.merge(sb,resp);

What is the expected output? What do you see instead?
The String in the resp Message isn't correct.

I hope this helps to improve this usefull Util.

What steps will reproduce the problem?
1. create an GzipOutputstream
2. print to that stream
3. print again, it fails since the stream is closed.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

Create a common interface for all format types so that they are easily 
injectable & switchable using various frameworks (like spring).  Some kind of 
factory would be useful as well.

Here is a patch to add support for the smile 
(http://wiki.fasterxml.com/SmileFormatSpec) format using the jackson processor. 
It closely matches the jsonformat, but adds a few more tests & works for 
extensions.

Please let me know if you would prefer to get this patch another way.

thanks

What steps will reproduce the problem?
1. define message with a field with default value.
2. populate a builder of that message, but don't set that field.
3. observe that json doesn't contain default value.

What is the expected output? What do you see instead?
should contain that field, but with default value set.


What version of the product are you using? On what operating system?
version 1.2, on a mac.

Please provide any additional information below.

What steps will reproduce the problem?
1. Call XmlFormat.merge(Readable input, Message.Builder builder) with an XML 
string

What is the expected output? What do you see instead?
Expect XML to be parsed. Instead I get an error about a missing "{" from the 
JsonFormat class.

What version of the product are you using? On what operating system?
1.1

Please provide any additional information below.

It seems that the code in XmlFormat is calling the following instead of just 
calling its own merge method:
        JsonFormat.merge(input, ExtensionRegistry.getEmptyRegistry(), builder);

1. set a String field with an escaped char
2. view the page in html and the chars appears escaped instead of actual char


What is the expected output? What do you see instead?
normal character, let the browser handle the encoding

JSON document:

{
                 "username":"検索jan5検索[email protected]"
                 ,"password":"12341234"
                 ,"display_name":"1231212"
                 ,"country":"US"
}

parses username field as: [“jan5”[email protected]]

HtmlFormat requires a unit test

What steps will reproduce the problem?

1. in .proto: 

message BigData { required bytes d = 1; }

2. in .java: 

BigData bd = BigData.newBuilder()
    .setD(ByteString.copyFrom(new byte[1024]))
    .build();

JsonFormat.merge(JsonFormat.printToString(bd), BigData.newBuilder());


What is the expected output? What do you see instead?

Saw StackOverflow exception:
Exception in thread "main" java.lang.StackOverflowError
        at java.util.regex.Pattern$CharProperty.match(Unknown Source)
        at java.util.regex.Pattern$Branch.match(Unknown Source)
        at java.util.regex.Pattern$GroupHead.match(Unknown Source)
        at java.util.regex.Pattern$Loop.match(Unknown Source)
        at java.util.regex.Pattern$GroupTail.match(Unknown Source)
        at java.util.regex.Pattern$BranchConn.match(Unknown Source)
        at java.util.regex.Pattern$CharProperty.match(Unknown Source)
        at java.util.regex.Pattern$Branch.match(Unknown Source)
        at java.util.regex.Pattern$GroupHead.match(Unknown Source)
        at java.util.regex.Pattern$Loop.match(Unknown Source)
        at java.util.regex.Pattern$GroupTail.match(Unknown Source)
        at java.util.regex.Pattern$BranchConn.match(Unknown Source)
......

What version of the product are you using? On what operating system?

protobuf-format-java-1.1 on Windows

Please provide any additional information below.

What steps will reproduce the problem?
1. printToJson from a Message
2.
3.

What is the expected output? What do you see instead?
Expected for a message
Person {
 optioanl string name = 1;
}

with the value name = test:

{ "name": "test" }

Actual:

{ "person": { "name": "test" }}


What version of the product are you using? On what operating system?
2.3.0 on osx

Please provide any additional information below.

This top-level wrapping isn't very json-esque but more like what you would see 
in xml. It should be possible to disable.

This may not be possible, but could the MessageLite interface be supported?

XmlFormat.print(Message, Appendable)
+ XmlFormat.print(MessageLite, Appendable)

What steps will reproduce the problem?
1. Define any message with a string field, e.g.
message TestMessage {
    required string test_string = 1;
}
2. Create an instance of this message with a string containing a single quote, 
e.g.

TestMessage msg = TestMessage.newBuilder().setTestString( "a string with 
'single quotes'").build()
String serialized = JsonFormat.printToString(msg);

3. The serialized string will look like the following:
{ "test_string":"a string with \'single quotes\'" }

But according to the JSON spec (http://www.json.org/), a single quote is not a 
valid symbol to follow a reverse solidus (single quotes don't need to be 
escaped at all in JSON). Trying to deserialize this through any 
standards-compilant JSON parser (e.g. Jackson) will result an error related to 
the fact that \' is not a valid character in JSON-encoded data.

What is the expected output? What do you see instead?
For syntactically-valid / standards-compliant JSON, I would expect to see 
instead
{ "test_string":"a string with 'single quotes'" }

[note the lack of reverse solidus before the single quotes.]

What version of the product are you using? On what operating system?
OS X 10.6, library version 1.1, Jackson 1.5.4

Thanks.

What steps will reproduce the problem?
1. Create a proto and add unknown fields (using UnknownFieldSet.Builder) to it 
with wire type fixed32 or fixed64
2. Serialize the proto to JSON using JsonFormat.printToString()
3. Inspect the output string

What is the expected output? What do you see instead?

The output will contain:

"9000": [0x00002328], "9001": [0x0000000000002329]

In this case, there is an unknown fixed32 field with tag 9000 and value 9000 (= 
0x2328), and an unknown fixed64 field with tag 9001 and value 9001 (= 0x2329).

However, this output is not valid JSON. Numbers in JSON can only be represented 
in decimal form.

If we attempt to parse this output using another JSON library (such as GSON), 
it may fail outright due to the bad syntax, but even if it is lenient enough to 
perform the conversion, there may not be a way to determine whether the value 
in the input stream was "0x" + 8 digits or "0x" + 16 digits. In such a case, 
there would be no way to determine which wire type was originally used to store 
the value (fixed32 or fixed64), which means we can't reconstruct the proto in a 
way that would allow us to reserialize it in binary form without loss.

There are important use cases for representing unknown fields losslessly. For 
example a server may hand an object to a client that only knows about an older 
version of the proto. Then the client may update some fields it knows about, 
and hand the object back to the server. Fields of the proto that are unknown to 
the client should not be disturbed. The ability to handle this kind of scenario 
seamlessly, without the need to synchronously update all consumers of a proto 
definition, is a key advantage of protocol buffers.

In effect, serialization of unknown fields is only lossless when they will be 
read by another instance of protobuf-java-format. This negates the value of 
using a standard format like JSON. Instead, the values should be encoded in 
such a way that only standard JSON constructs are used, and the wire type is 
preserved, for example:

"9000":[{"fixed32":9000},{"fixed64":9001}]

What version of the product are you using? On what operating system?

1.2.

What steps will reproduce the problem?
1. Create an extensible message
2. Create an extension
3. Instantiate both and initialize the extensible instance with an extension 
instance
4. Serialize to JSON.

What is the expected output? What do you see instead?

Using this .proto descriptor:

message Foo {
  required uint32 foo = 1;
  extensions 100 to max;
}

message Bar {
  required uint32 bar = 1;
  extend Foo {
    optional Bar bar_ext = 100;
  }
}

Instantiate and serialize:

FooBar.Bar bar = FooBar.Bar.newBuilder().setBar(42).build();
JsonFormat.printToString(
  FooBar.Foo.newBuilder()
    .setFoo(4)
    .setExtension(FooBar.Bar.barExt, 
       FooBar.Bar.newBuilder()
        .setBar(42).build()
    )
);

Expected:
{"foo": 4,"Bar.bar_ext": {"bar": 42}}

Actual:
{"foo": 4,["Bar.bar_ext"]: {"bar": 42}}

What version of the product are you using? On what operating system?

protobuf-java-format-1.1

Please provide any additional information below.

What steps will reproduce the problem?
1. Create a builder and set a field on an inner message to a non-default 
value
2. Create the appropriate XML that you would use to populate the builder 
above with.  Leave the field changed above in it's default state - which 
means that the field does not appear in the XML.
3. Invoke merge(xml, builder) with the two and notice that a merge does not 
occur because the inner message overwrites the one on the builder.  
Consequently, the value in the builder that you wished to "merge" with the 
XML is gone.

What is the expected output? What do you see instead?  I expect that the 
values from the XML are merged into the builder - with xml values 
overwriting those that are set in the builder.  This behavior is consistent 
with the mergefrom methods in PB.  Instead, the XML is serialized into a PB 
, destroying any prior PB state that you wanted merged in.

Admittedly, this may not be the intended functionality - but it would have 
been nice to have named the method appropriately if it is not.


What version of the product are you using? On what operating system?
1.1, JDK 1.6, Windows

Please provide any additional information below.

Hi,

I was wondering why a json parser like jackson was not used for json 
coding/decoding.  I wrote a simple codec for json which uses jackson as the 
underlying framework and found that the performance for json serialization 
almost doubled.  The implementation is quite basic and lacks supports for 
unknownfieldsets. 
Would appreciate a feedback since I just started out with protobuf.

I see that you have taken away all of my permissions to be able to contribute 
to the project without providing any notice or explanation.  Could you please 
fix the permissions so that I could continue to contribute!

Thanks,
Alex

You have the JSON - would be interesting to be able to generate the IDL/schema 
and service bindings for Avro support.

What steps will reproduce the problem?
1.Create a POJO class with an user defined class's object as instance member
2.Instantiate this pojo object with the member object's value as null
3.Convert this POJO to json string and invoke JsonFormat.merge()

What is the expected output? What do you see instead?
Json string should be properly mapped to its corresponding ProtoBuf object. But 
Getting exception
           Exception in thread "main" com.googlecode.protobuf.format.JsonFormat$ParseException: 1:16: Expected "{".
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.parseException(JsonFormat.java:765)
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.consume(JsonFormat.java:530)
    at com.googlecode.protobuf.format.JsonFormat.handleObject(JsonFormat.java:1128)
    at com.googlecode.protobuf.format.JsonFormat.handleValue(JsonFormat.java:1004)
    at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:952)
    at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:870)
    at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:818)
    at com.ms.JSONToPB.main(JSONToPB.java:48)

What version of the product are you using? On what operating system?
 protobuf-java-format - 1.2. OS - Windows 7

Please provide any additional information below.

main function I used
--------------------
public static void main(String[] args)(){
        Source sObj = new Source();
    sObj.setMemberClass(null);
    SourceProto.Source.Builder builder = SourceProto.Source.newBuilder();
    ObjectMapper objMapper = new ObjectMapper();        
    String jsonString = objMapper.writeValueAsString(sObj);
    JsonFormat.merge(jsonString, builder);
    SourceProto.Source obj = builder.build();
}

Source.class
------------
public class Source{

    private MemberClass memberClass;    

    public MemberClass getMemberClass() {
        return memberClass;
    }

    public void setMemberClass(MemberClass memberClass) {
        this.memberClass = memberClass;
    }
}
MemberClass.class
------------------
public class MemberClass {
    private int num;

    public int getNum() {
        return num;
    }

    public void setNum(int num) {
        this.num = num;
    }   
}

If we keep null value for primitive type, it is parsed properly. For 
non-primitive types, error is thrown

I went through the code and modified the JsonFormat.java at line number : 1123. 
I added the following code

        if("null".equals(tokenizer.currentToken)){
            tokenizer.consume("null");
            return null;
        }

With this null check, issue is resolved. Please check whether this is the 
proper resolution

I see that you have taken away all of my permissions to be able to contribute 
to the project without providing any notice or explanation.  Could you please 
fix the permissions so that I could continue to contribute!

Thanks,
Alex

Add ability to de-serialize Protobuf objects from XML source.

Please publish the JAR as Maven artifact (ideally in the central repository 
but a Google Code one would do)

What steps will reproduce the problem?
1.include the current 1.2 module as a build dependency for a java 1.5 jar
2.compile

What is the expected output? 
no complaints and a working build

What do you see instead?
/WorkSpace/p4/modules/serve/main/data-transform/src/main/java/com/comp/data/tran
sform/protocolbuffer/ProtocolBufferToJsonTransformer.java:[5,-1] cannot access 
com.googlecode.protobuf.format.JsonFormat
bad class file: 
/media/WorkSpace/repo/com/googlecode/protobuf-java-format/protobuf-java-format/1
.2/protobuf-java-format-1.2.jar(com/googlecode/protobuf/format/JsonFormat.class)
class file has wrong version 50.0, should be 49.0


What version of the product are you using? On what operating system?
1.2, Linux, Java 1.5

Please provide any additional information below.
I was able to fix this by rebuilding the jar with the following changes to the 
pom.xml file.  No code changes were required.

 <plugins>
   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
     <source>1.5</source>
     <target>1.5</target>
    </configuration>
   </plugin>
 </plugins>

What steps will reproduce the problem?
1. Create any protobuf where a String that consists only of spaces is valid
2. Transform it to xml using this tool
3. The field will be incorrect in the XML

What is the expected output? What do you see instead?

I see the first characters of the closing tag - I expect to see the spaces.
What version of the product are you using? On what operating system?
1.1, Java 1.6.0_18, Windows 7 

Please provide any additional information below.

Elements are allowed to have significant spaces in their text.  The method 
nextToken(), however, will eliminate any spaces following a valid token - 
including the closing bracket of an opening element tag.  So, if I have 
something like this:
  <AnElementWithSpaces> </AnElementWithSpaces>
The tokenizer will correctly recognize the closing bracket of 
<AnElementWithSpaces>, but will remove the following (significant) spaces.
The software should remove insignificant spaces and retain the others.  A 
quick look at the code suggests that the only time this is necessary is in 
the mergeField() method, where it first consumes the opening bracket and 
then consumes the identifier.  Consuming the identifier should not delete 
the significant spaces.

amoffetATgmailDOTcom

What steps will reproduce the problem?
1. Create a JSON string with field Foo that has a double as a key
2. use JsonFormat.merge with a Builder that does not have Foo defined

What is the expected output? What do you see instead?
The call to JsonFormat.merge fails with a message like:
Parseexception: 1:14: Couldn't parse integer: For input string: "0.95"

What version of the product are you using? On what operating system?
1.2

Please provide any additional information below.
Defining Foo as a double in the protobuf is a workaround, but I think it would 
be better if support for double was added to handleMissingField()

RE: Issue-35

What steps will reproduce the problem?

1. download pjfeoi.tgz
2. extract it (tar -xzvf pjfeoi.tgz)
3. cd pjf_empty_object
4. make clean && make && make run


What is the expected output? What do you see instead?

expected:

textJson: {"text":"foo"}
textPdu: {"text": "foo"}
authAckJson: {"authAck":{}}
authAckPdu: {"authAck": {}}

actual:

textJson: {"text":"foo"}
textPdu: {"text": "foo"}
authAckJson: {"authAck":{}}
com.googlecode.protobuf.format.JsonFormat$ParseException: 1:13: Expected 
identifier. -}
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.parseException(JsonFormat.java:765)
    at com.googlecode.protobuf.format.JsonFormat$Tokenizer.consumeIdentifier(JsonFormat.java:575)
    at com.googlecode.protobuf.format.JsonFormat.handleMissingField(JsonFormat.java:970)
    at com.googlecode.protobuf.format.JsonFormat.mergeField(JsonFormat.java:936)
    at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:870)
    at com.googlecode.protobuf.format.JsonFormat.merge(JsonFormat.java:818)
    at com.chrisdew.Main.main(Main.java:26)
authAckPdu: {}


What version of the product are you using? On what operating system?

$ ls libs/protobuf*
libs/protobuf-2.5.0.jar  libs/protobuf-java-format-1.2.jar

$ java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=12.04
DISTRIB_CODENAME=precise
DISTRIB_DESCRIPTION="Ubuntu 12.04.2 LTS"

$ uname -a
Linux chris-work 3.2.0-45-generic #70-Ubuntu SMP Wed May 29 20:12:06 UTC 2013 
x86_64 x86_64 x86_64 GNU/Linux


Please provide any additional information below.

pjfeoi.proto:

package com.chrisdew.pjfeoi;

message Pdu {
  optional string  text     = 4;
  optional AuthAck auth_ack = 5;
}
message AuthAck {
}


Main.java:

package com.chrisdew;

import com.google.protobuf.UninitializedMessageException;
import com.googlecode.protobuf.format.JsonFormat;
import com.chrisdew.pjfeoi.Pjfeoi;

public class Main {
        public static void main(String[] args) {
                String textJson = "{\"text\":\"foo\"}";
                System.out.println("textJson: " + textJson);

                Pjfeoi.Pdu.Builder builderForText = Pjfeoi.Pdu.newBuilder();
                try {
                        JsonFormat.merge(textJson, builderForText);
                } catch (JsonFormat.ParseException e) {
                        e.printStackTrace();
                }
                Pjfeoi.Pdu textPdu = builderForText.build();
                System.out.println("textPdu: " + JsonFormat.printToString(textPdu));

                String authAckJson = "{\"authAck\":{}}";
                System.out.println("authAckJson: " + authAckJson);

                Pjfeoi.Pdu.Builder builderForAuthAck = Pjfeoi.Pdu.newBuilder();
                try {
                        JsonFormat.merge(authAckJson, builderForAuthAck);
                } catch (JsonFormat.ParseException e) {
                        e.printStackTrace();
                }
                Pjfeoi.Pdu authAckPdu = builderForAuthAck.build();
                System.out.println("authAckPdu: " + JsonFormat.printToString(authAckPdu));
        }
}

What steps will reproduce the problem?
1. A string field in a message contains characters like à, ò, è, é (etc)
2. Look at the output of String jsonFormat =
jsonFormat.printToString(someProto);

What is the expected output? What do you see instead?
I want to see the original characters, not \xxx octal sequences.

What version of the product are you using? On what operating system?
protobuf-format-java-1.1.jar

Please provide any additional information below.
I tried to modify the method escapeBytes in JsonFormat.java but I can't see
my original characters.

The wiki contains the following path to use as a maven repository:

http://protobuf-java-format.googlecode.com/svn/trunk/

The path to the protobuf-format-java artifact looks like this:

com/google/protobuf/protobuf-java-format/1.1.1/

But the artifact is not named protobuf-java-format, it's named 
protobuf-format-java, which causes maven to be unable to resolve the pom and 
jar correctly when added as a dependency as follows:

<dependency>
  <groupId>com.google.protobuf</groupId>
  <artifactId>protobuf-format-java</artifactId>
  <version>1.1.1</version>
</dependency>

What steps will reproduce the problem?
1. Create a valid XML file representing a protobuf object. You could call 
XmlFormat.print on a protobuf object, for instance.
2. Add a valid XML comment to the file
3. Call XmlFormat.merge with that file and you'll get an exception

What is the expected output? What do you see instead?
I'd expect it to create the protobuf object. Instead it causes an exception of 
the form:

com.google.protobuf.XmlFormat$ParseException: 4:4: Expected identifier. -!
 at com.google.protobuf.XmlFormat$Tokenizer.parseException(XmlFormat.java:662)

What version of the product are you using? On what operating system?
1.1 on Mac OS X.

Please provide any additional information below.

What steps will reproduce the problem?
1.  Snippet from my test program:
        //  javascript in browser gets byte string embedded in larger message, using ajax.  if the
        //  data array has 0x07 or 0x0b, browser throws exception for the ajax thread.  please see the
        //  standard ECMA-262, at "http://www.ecma-international.org/publications/standards/Ecma-262.htm",
        //  section 15.12.1.1, titled "The JSON Lexical Grammar".  That section clearly states:
        //
        //      JSONEscapeCharacter  :: one of       " / \ b f n r t
        //
        //  There is no mention of "\a" or "\v" as legal escape shortcuts in a JSON string literal.
        //  

        //  snippet from .proto file ..
        //  
        //    required uint64        data_id = 1;
        //    required uint64        event_id = 2;
        //    optional bytes         data = 3;
        //  

        //  note .. in C-language (or, at least, Microsoft C compilers), there are extra
        //  escapes allowed:
        //
        //      \a      ==>  0x07       "alert"         (ascii BEL)
        //      \v      ==>  0x0b       "vertical tab"  (ascii VT)
        //
        byte[]  caseAbytes =  { 0x06, 0x07, 0x08 };     //  expect "\u0006\u0007\b", get "\u0006\a\b"
        byte[]  caseVbytes =  { 0x0a, 0x0b, 0x0c };     //  expect "\n\u0016\f", get "\n\v\f"

        ByteString  caseA =  ByteString.copyFrom( caseAbytes );
        ByteString  caseV =  ByteString.copyFrom( caseVbytes );

        //  The field "output_event_data" is a submessage of type EventData.  We
        //  want to alter the field "data" in that submessage.
        //
        //        at ..
        //
        DynamicMessage.Builder  editorProto =  parsedProto.newBuilder( parsedProto );
        DynamicMessage.Builder  editorOED =  editorProto.newBuilderForField( fieldOED );

        fieldOED_data_id =  fieldOED.getMessageType().findFieldByName( "data_id" );     
        fieldOED_event_id =  fieldOED.getMessageType().findFieldByName( "event_id" );
        fieldOED_data =  fieldOED.getMessageType().findFieldByName( "data" );

        try
        {
            Message  patchProto =  null;

            editorOED.setField( fieldOED_data_id, new Long(1) );
            editorOED.setField( fieldOED_event_id, new Long(2) );

            editorOED.setField( fieldOED_data, caseA );
            patchProto =  editorOED.build();

            if  (DUMP_CASE_A)
            {
                String  formatCaseA =  JsonFormat.printToString( patchProto );
                System.out.println( "illegal escape, demo case \\a: " + formatCaseA );
                System.out.println( "  ** should have been: \\u0006\\u0007\\b   **" );
            }

            editorOED =  editorProto.newBuilderForField( fieldOED );
            editorOED.setField( fieldOED_data_id, new Long(3) );
            editorOED.setField( fieldOED_event_id, new Long(4) );

            editorOED.setField( fieldOED_data, caseV );
            patchProto =  editorOED.build();

            if  (DUMP_CASE_V)
            {
                String  formatCaseV =  JsonFormat.printToString( patchProto );
                System.out.println( "illegal escape, demo case \\v: " + formatCaseV );
                System.out.println( "  ** should have been: \\n\\u0011\\f   **" );
            }
        }
        catch  (Exception  e)
        {
            e.printStackTrace();
        }

2. Wrap the snippet above in a compilable java source and run it.

The println() calls and comments document what should happen.
Impact if using ajax to feed ByteString to browser is that a browser such as 
Firefox complains about "invalid escaped character".

Using 1.2 release on linux 64-bit Centos 6.2 build.

Please provide any additional information below:

You can see in the source at 
"http://code.google.com/p/protobuf-java-format/source/browse/trunk/src/main/java
/com/googlecode/protobuf/format/JsonFormat.java",  line 1061, method 
escapeBytes() that someone ignored a strong hint about \a and \v and went ahead 
to (improperly) escape them for JSON output.  

1. Define any message with a bytes field.
2. Create an instance of that message with that field set to a value that 
contains bytes 0x07 or 0x0b
3. Call JsonFormat.printToString. The generated JSON will contain a string 
value for this field with the tokens \a and \v in this value. This is invalid 
JSON according to the JSON spec (http://www.json.org/) because 'a' and 'v' are 
not amongst the characters that are allowed to follow a reverse solidus. No 
standards-compliant JSON parser (e.g. Jackson) or validator (e.g. 
www.jsonlint.org) will parse this JSON.

These escapes are explicitly added by the method JsonFormat.escapeBytes(). It's 
unclear why \a and \v need special handling rather than just being turned into 
unicode characters.

Environment:
* JsonFormat 1.1 (though the head of the current source (r. 34) currently has 
this issue as well)
* OS X 10.6.4
* Java:
** java version "1.6.0_20"
** Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
** Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)
* Jackson 1.5.4 (though any other standards-compliant parser will fail to parse 
this as well, e.g. www.jsonlint.org)

JsonFormat performs a build instead of buildPartial. As such if required field 
is missing in base object then it successfully perform merge, however if some 
nested message's required field is missing, merge causes exception. Changing 
build to buildPartial at all the 2 places inside this file should work.

Unknown fields are printed in a single element and not repeated.

What steps will reproduce the problem?
1. Produce any protobuf that contains a date like 11 Mar 2009 as a value
2. Convert it to XML
3. XMLFormat will throw an exception

What is the expected output? What do you see instead?

I expect to see a date, without any errors
What version of the product are you using? On what operating system?

1.1, java 1.6.0, Windows 7
Please provide any additional information below.

The regular expression (TOKEN) will group a number by itself, before 
grouping a date like 11 March 2009 together.  Subsequently, the tokenizer 
will look for an end tag and not find one.

amoffetATgmailDOTcom

What steps will reproduce the problem?
1. Create a collection of Message instances
2. Use JsonFormat to serialize them

What is the expected output? What do you see instead?

The expected output is an array of Messages:
[ {"name":"msg1"}, {"name":"msg2"}, {"name":"msg3"} ]

Since there is no support for serializing a collection of Message
instances, the calling code needs to "manually" open and close the array
brakets.

Ideally, the calling code would call a method with the following signature:

JsonFormat.print(Iterable<Message> messages, Appendable a);

What version of the product are you using? On what operating system?
1.1

Please provide any additional information below.

Or is the pattern to create a Message type that has a repeatable field of
other Message types?

message Foo {
  required string name = 1;
}

message Foos {
  repeatable Foo foos = 1;
}

What steps will reproduce the problem?
1. test.proto

message TestItem{
     required string  name  = 1;
}
2. Java code

Testproto.TestItem.Builder itemBuilder =  Testproto.TestItem.newBuilder();
itemBuilder.setName("name with spaces");
String itemTxt = XmlFormat.printToString(itemBuilder.build());
itemBuilder = Testproto.TestItem.newBuilder();
XmlFormat.merge(itemTxt,  itemBuilder);
System.out.println("success");


What is the expected output? 
success

What do you see instead?
com.google.protobuf.XmlFormat$ParseException: 1:27: Expected ">".


What version of the product are you using? On what operating system?
protobuf 2.2.0
protobuf-java-format 1.1
OS: win32

Please provide any additional information below.
Tokenizer works not correctly.
Way of fix in attached patch.

What steps will reproduce the problem?
1. Serialize a object with bytes "P\001"
2. Try to deserialize it (XML)
3. Cannot deserialize

A little fix for that (bytes are still readable, plus serialize by default in 
hex):

--- XmlFormat.java  2010-06-03 17:06:18.000000000 +0100
+++ XmlFormat.java  2010-10-14 15:46:09.792528214 +0100
@@ -39,6 +39,7 @@

 import com.google.protobuf.Descriptors.EnumValueDescriptor;
 import com.google.protobuf.Descriptors.FieldDescriptor;
+import com.google.protobuf.util.*;

 /**
  * Provide ascii text parsing and formatting support for proto2 instances. The implementation
@@ -361,10 +362,11 @@
         private static final Pattern WHITESPACE =
           Pattern.compile("(\\s|(#.*$))++", Pattern.MULTILINE);
         private static final Pattern TOKEN = Pattern.compile(
-          "extension|" + "[a-zA-Z_\\s;@][0-9a-zA-Z_\\s;@+-]*+|" +        // an 
identifier with special handling for 'extension'
+          "extension|" +                                // special handling 
for 'extension'
+          "[0-9a-zA-Z\\\\\\\"\\\']++|" +                // a \000 byte 
sequence for bytes handling
+          "[a-zA-Z_\\s;@][0-9a-zA-Z_\\s;@+-]*+|" +      // an identifier
           "[.]?[0-9+-][0-9a-zA-Z_.+-]*+|" +             // a number
           "</|" +                                       // an '</' closing element marker
-          "[\\\\0-9]++|" +                              // a \000 byte 
sequence for bytes handling
           "\"([^\"\n\\\\]|\\\\.)*+(\"|\\\\?$)|" +       // a double-quoted string
           "\'([^\'\n\\\\]|\\\\.)*+(\'|\\\\?$)",         // a single-quoted string
           Pattern.MULTILINE);
@@ -1041,13 +1043,11 @@
                     builder.append("\\\"");
                     break;
                 default:
-                    if (b >= 0x20) {
+                    if ((b >= 0x30 && b <= 0x39) || (b >= 0x41 && b <= 0x5A) 
|| (b >= 0x61 && b <= 0x7A)) {
                         builder.append((char) b);
                     } else {
-                        builder.append('\\');
-                        builder.append((char) ('0' + ((b >>> 6) & 3)));
-                        builder.append((char) ('0' + ((b >>> 3) & 7)));
-                        builder.append((char) ('0' + (b & 7)));
+                        builder.append("\\x");
+                        builder.append(HexUtils.getHexString(b, 2));
                     }
                     break;
             }