joyrex2001 / protobuf-perlxs Goto Github PK

I have .proto files that I want to use with a mobile client talking to a
perl server. The mobile code needs to be compiled with optimize_for =
LITE_RUNTIME but the generated wrappers are wrong as they don't take this
into account as they unconditionally try to link in calls to non-lite methods.

The workaround is to preprocess the .protos in Makefile.PL to strip out the
LITE_RUNTIME for the perl target.

Would be nice to either make the perl target also understand LITE_RUNTIME.

(though I think having LITE_RUNTIME be a property of a .proto rather than a
protoc argument is probably a sub-optimal design).

What steps will reproduce the problem?
1. Use a nested message construct.
2. After compilation and use, do an ISA on the nested message and the
   class is that of the nesting class, not the nested class.

What is the expected output? What do you see instead?

Here's a proto that will show the issue:

------------------
package Report;

message Statistic {
    optional string key = 1;
    optional string val = 2;
}

message Query {
    optional string    name  = 1;
    repeated Statistic stats = 2;
}
------------------

What version of the product are you using? On what operating system?

0.2 build on Linux 2.6.21/gcc 4.2.4/perl 5.8.8

Please provide any additional information below.

I attached a patch - simple fix. Thanks for coding this up.

-John

What steps will reproduce the problem?
1. ./configure --with-protobuf=/usr/local/
2. make

What is the expected output? What do you see instead?

  Making all in src
  g++  -g -O2  -L/usr/local/lib -o protoxs  perlxs_generator.o 
perlxs_helpers.o main.o -lprotoc 
  Undefined symbols:
    "google::protobuf::FieldDescriptor::kTypeToCppTypeMap", referenced from:
    (and lots, lots more undefined symbols - see attached file)

It looks like it can't find libprotoc.a but it's installed there in my
/usr/local/lib

What version of the product are you using? On what operating system?

  protobuf 2.0.2, protobuf-perlxs-0.2, Mac OXS 10.5.4

Please provide any additional information below.

  typescript attached showing build session

I noticed that strings with embedded NUL ("\0") characters in them were
being truncated when using either the HASHREF form of the ->new()
constructor or the ->set_FIELD() accessor for the generated XS class.  The
problem appears to be in the conversion of the char * type to the string
type without referencing the length of the char * array.  I've included a
patch that performs the necessary steps to extract the length of the char *
type from the SV pointer and use both values when constructing string
types.  Please note that my experience with both XS and C++ is extremely
limited, so the patch is probably quite sub-optimal and is provided as a
proof-of-concept.

You can test this issue with something like the following:

[person.proto]
message Person {
  required string name  = 1;
  required bytes  image = 2;
}

[person.pl]
use Person;

my $p = Person->new({ name => "jason\0hord" });
$p->set_image( "binary\0goop" );

print
  'name:  ', $p->name, "\n",
  'image: ', $p->image, "\n",
  join( ' ', unpack '(H2)*', $p->pack ), "\n"
;

[pre-patch output]
name:  jason
image: binary
0a 05 6a 61 73 6f 6e 12 06 62 69 6e 61 72 79

[post-patch output]
name:  jason^@hord
image: binary^@goop
0a 0a 6a 61 73 6f 6e 00 68 6f 72 64 12 0b 62 69 6e 61 72 79 00 67 6f 6f 70

What steps will reproduce the problem?

1. Patch protobuf-perlxs-1.1/examples/embedded with the following:
Index: examples/embedded/test.pl
===================================================================
--- examples/embedded/test.pl   (revision 191663)
+++ examples/embedded/test.pl   (working copy)
@@ -84,6 +84,8 @@
     my $packed  = $error->pack;
     my $length  = length($packed);
     my $elapsed = tv_interval($now);
+    $error->clear();
+    $error->unpack($packed);

     print "Built/packed error of $length bytes in $elapsed seconds\n";

@@ -116,6 +118,8 @@
     my $packed  = $error->pack;
     my $length  = length($packed);
     my $elapsed = tv_interval($now);
+    $error->clear();
+    $error->unpack($packed);

     print "Built/packed error of $length bytes in $elapsed seconds\n";

2. perl Makefile.PL && make && make test

What is the expected output? What do you see instead?
I see output like this:

        Built/packed error of 141 bytes in 6.8e-05 seconds
        $VAR1 = {
                  'datetime' => 1415276412,
                  'severity' => 3,
                  'message' => 'Here iing"

        test'
                };
        Built/packed error of 141 bytes in 6.2e-05 seconds
        $VAR1 = {
                  'message' => 'Here iing"

        test',
                  'datetime' => 1415276412,
                  'severity' => 3
                };

instead of

        Built/packed error of 173 bytes in 7.7e-05 seconds
        $VAR1 = {
                  'pid' => 29231,
                  'message' => 'Here is a warning',
                  'trace' => [
                               {
                                 'file' => 'test.pl',
                                 'line' => 51
                               },
                               {
                                 'line' => 43,
                                 'file' => 'test.pl'
                               },
                               {
                                 'line' => 39,
                                 'file' => 'test.pl'
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 35
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 31
                               },
                               {
                                 'line' => 27,
                                 'file' => 'test.pl'
                               },
                               {
                                 'line' => 23,
                                 'file' => 'test.pl'
                               },
                               {
                                 'line' => 19,
                                 'file' => 'test.pl'
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 15
                               },
                               {
                                 'line' => 11,
                                 'file' => 'test.pl'
                               }
                             ],
                  'hostname' => 'spider-man',
                  'severity' => 3,
                  'datetime' => 1415276538
                };
        Built/packed error of 173 bytes in 6.1e-05 seconds
        $VAR1 = {
                  'pid' => 29231,
                  'message' => 'Here is a warning',
                  'trace' => [
                               {
                                 'file' => 'test.pl',
                                 'line' => 51
                               },
                               {
                                 'line' => 43,
                                 'file' => 'test.pl'
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 39
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 35
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 31
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 27
                               },
                               {
                                 'line' => 23,
                                 'file' => 'test.pl'
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 19
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 15
                               },
                               {
                                 'file' => 'test.pl',
                                 'line' => 12
                               }
                             ],
                  'datetime' => 1415276538,
                  'severity' => 3,
                  'hostname' => 'spider-man'
                };

What version of the product are you using? On what operating system?
- protobuf-perlxs-1.1
- Ubuntu 14.10
- $ perl -v

        This is perl 5, version 20, subversion 1 (v5.20.1) built for x86_64-linux-gnu-thread-multi
        (with 37 registered patches, see perl -V for more detail)

Please provide any additional information below.
I fixed this by that patch:
Index: src/google/protobuf/compiler/perlxs/perlxs_generator.cc
===================================================================
--- src/google/protobuf/compiler/perlxs/perlxs_generator.cc (revision 191667)
+++ src/google/protobuf/compiler/perlxs/perlxs_generator.cc (working copy)
@@ -123,21 +123,19 @@
   printer.Print("class $base$_OutputStream :\n"
        "  public google::protobuf::io::ZeroCopyOutputStream {\n"
        "public:\n"
+       "  explicit $base$_OutputStream(SV * sv) : sv_(sv) {}\n"
-       "  explicit $base$_OutputStream(SV * sv) :\n"
-       "  sv_(sv), len_(0) {}\n"
        "  ~$base$_OutputStream() {}\n"
        "\n"
        "  bool Next(void** data, int* size)\n"
        "  {\n"
+       "    STRLEN byteCount = SvCUR(sv_);\n"
+       "    STRLEN available = SvLEN(sv_) - byteCount;\n"
+       "    if (available <= 16)\n"
+       "    {\n"
+       "      SvGROW(sv_, 2 * byteCount);\n"
+       "      available = SvLEN(sv_) - byteCount;\n"
+       "    }\n"
+       "    SvCUR_set(sv_, byteCount + available);\n"
+       "    *data = SvPVX(sv_) + byteCount;\n"
+       "    *size = available;\n"
-       "    STRLEN nlen = len_ << 1;\n"
-       "\n"
-       "    if ( nlen < 16 ) nlen = 16;\n"
-       "    SvGROW(sv_, nlen);\n"
-       "    *data = SvEND(sv_) + len_;\n"
-       "    *size = SvLEN(sv_) - len_;\n"
-       "    len_ = nlen;\n"
        "\n"
        "    return true;\n"
        "  }\n"
@@ -147,7 +145,11 @@
        "    SvCUR_set(sv_, SvLEN(sv_) - count);\n"
        "  }\n"
        "\n"
+       "  void Sync() {}\n"
-       "  void Sync() {\n"
-       "    if ( SvCUR(sv_) == 0 ) {\n"
-       "      SvCUR_set(sv_, len_);\n"
-       "    }\n"
-       "  }\n"
        "\n"
        "  int64_t ByteCount() const\n"
        "  {\n"
@@ -156,6 +158,7 @@
        "\n"
        "private:\n"
        "  SV * sv_;\n"
-       "  STRLEN len_;\n"
        "\n"
        "  GOOGLE_DISALLOW_EVIL_CONSTRUCTORS($base$_OutputStream);\n"
        "};\n"

Looks like a simple typo in google/protobuf/compiler/perlxs/perlxs_generator.cc:

"discard_unkown_fields(svTHIS)\n"

What steps will reproduce the problem?
1. Create a protobuf message with incomplete data in ->new()
2. Call the usual message->pack() 
3. libprotobuf FATAL google/protobuf/message.cc:212] CHECK failed:

What is the expected output? What do you see instead?

When I wrap the message->pack() routing inside an eval {} block, I expect
to be able to catch the exception.  Instead, it seems that the exception
thrown is somehow bypassing the eval {} block.

What version of the product are you using? On what operating system?

protobuf-perlxs v0.6
perl, v5.8.8 built for i486-linux-gnu-thread-multi

Please provide any additional information below.

Here's an example code:

Using protobuf:
message Person {
   required int32 id = 1;
   required string name = 2;
   optional string email = 3;
}

----
Perl Code:

use Person;
my $person = Person->new({id => 1});

eval {
    $person->pack();
};
----

The exception should be handled.  Please advise on how to gracefully handle
these exceptions from XS code.  Thanks.

-- Darien

What steps will reproduce the problem?
"make test"

What is the expected output? What do you see instead?
I see: *** Signal 11

What version of the product are you using? On what operating system?
FreeBSD 9.0 using any self-compiled unthreaded perl (tried 5.8.9, 5.10.x, 
5.14.2, 5.16.2) or the system perl (5.12.4, also unthreaded).

Please provide any additional information below.
With a threaded perl "make test" works.

After doing some experimentation I found an issue trying to initialize
repeated elements when passing a HASHREF to ->new() for a protoxs generated
class.  I was able to track it down to an off-by-one condition in the for()
loop that initializes the repeated item in the from_hashref() helper.  I've
attached a patch that solves the problem on my end.

What steps will reproduce the problem?

Installed for perl 5.14.1, trying example 'big':
 perl Makefile.PL - ok
 make - ok
 make test - error:

Not a CODE reference at 
/usr/local/lib/perl5/5.14.1/i686-linux-thread-multi/DynaLoader.pm line 213.
END failed--call queue aborted at blib/lib/Big/Message.pm line 213.
Compilation failed in require at test.pl line 9.
BEGIN failed--compilation aborted at test.pl line 9.

But with ubuntu's perl 5.10.1 works good:

0.017285 seconds to set 10 blob fields
0.019572 seconds to pack message of 20971570 bytes
0.022064 seconds to unpack packed message
0.045516 seconds to freeze message of 20971600 bytes
0.029589 seconds to thaw frozen hashref



What version of the product are you using? On what operating system?

Ubuntu 11.04, perl 5.14.1 (self-built).

I found this out when my message had a 'length' field, and it took me a little 
while to see what was going on. 

Not sure of the best way to handle this:

1) print a warning from protoxs when a message field matches a reserved method 
name.
2) along with the above, create the method with a trailing undescrore (protobuf 
will do this with some already, i.e. namespace_)
3) do nothing, thats just how it goes.

What steps will reproduce the problem?
1. Add next strings to examples/types/test.pl:
$msg1->add_rep_message(new Protobuf::Types::Message({ t_string => "message 1" 
}));
$msg1->add_rep_message(new Protobuf::Types::Message({ t_string => "message 2" 
}));
$msg1->add_rep_message(new Protobuf::Types::Message({ t_string => "message 3" 
}));
$msg1->add_rep_message(new Protobuf::Types::Message({ t_string => "message 4" 
}));
$msg1->add_rep_message(new Protobuf::Types::Message({ t_string => "message 5" 
}));

2. Add more 'print'-s:
print Dumper $msg1->rep_message(0)->to_hashref();
print Dumper $msg1->rep_message(1)->to_hashref();
print Dumper $msg1->rep_message(2)->to_hashref();
print Dumper $msg1->rep_message(3)->to_hashref();
print Dumper $msg1->rep_message(4)->to_hashref();

3. Run 'make test'

What is the expected output? What do you see instead?

I see this:

          'rep_message' => [
                             {
                               't_string' => 'message 5'
                             },
                             {
                               't_string' => 'message 5'
                             },
                             {
                               't_string' => 'message 5'
                             },
                             {
                               't_string' => 'message 5'
                             },
                             {
                               't_string' => 'message 5'
                             }
                           ],

All messages are the same due to bug in to_hashref() Types.xs:
...
      if ( msg0->rep_message_size() > 0 ) {
        AV * av0 = newAV();
        SV * sv0 = newRV_noinc((SV *)av0);

        for ( int i0 = 0; i0 < msg0->rep_message_size(); i0++ ) {
          ::Protobuf::Types_Message * msg2 = msg0->mutable_rep_message(i0);
          SV * svv1 = sv_2mortal( __Protobuf__Types_call_Method( svTHIS, "rep_message" ) );
          SV * sv1 = __Protobuf__Types_call_Method( svv1, "to_hashref" );

          av_push(av0, sv1);
        }
        hv_store(hv0, "rep_message", sizeof("rep_message") - 1, sv0, 0);
      }
...



What version of the product are you using? On what operating system?

Last version and last Ubuntu.


Please provide any additional information below.

What steps will reproduce the problem?
1. protoxs --cpp_out=. --out=. toto.proto
2.
3.

What is the expected output? What do you see instead?
Output files ;) => infinite loop, allocate all memory

I expect output xs files, with Foo->bar that can be undef


What version of the product are you using? On what operating system?
protoxs 1.1

Please provide any additional information below.

Try protoxs with this .proto

message Foo {
      repeated Foo bar = 14;
}

message OUTPUT_FORMAT {
      required Foo main = 3;
}