qitab / cl-protobufs Goto Github PK

Not sure why this is happening, but we can see running:
https://github.com/Slids/cl-protobufs/runs/1044380174?check_suite_focus=true
it's true.

Because I happen to want to run a grpc server written in Common Lisp on GNU/Linux aarch64 (arm64), I'm interested in what it would take to upstream support for this. Most notably, aarch64 is (by default) big endian.

Based on some cursory looking, it looks like only define-fixed-width-encoder needs to be implemented?

For implementing this, would it be acceptable to bring in a whole or parts of a library like swap-bytes? This particular library is available under the MIT license. The strategy could be then to:

• Add htonl/htonq equivalents that convert integers to protobuf wire format (little endian)
• Wrap fixed with integer accessors (e.g. sb-sys:sap-ref-N) to a call to those equivalents

On little-endian machines the conversion functions would become no-ops, and on big-endian machines it would call the internal swap-bytes implementation.

This library currently does not implement a native byte-swap routine for arm64 and would resort to a portable fallback, but a faster implementation using arm64 intrinsics could be upstreamed.

Building with SBCL 2.3.1.43-c2e7dcbd6 / ASDF 3.3.5 for quicklisp dist creation.

Trying to build commit id 6300379

cl-protobufs fails to build with the following error:

; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE FIELD-DESCRIPTOR-PROTO ...))
;   The value TYPE-DOUBLE is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE FILE-OPTIONS ...))
;   The value SPEED is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE FIELD-OPTIONS ...))
;   The value STRING is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE METHOD-OPTIONS ...))
;   The value IDEMPOTENCY-UNKNOWN is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
Unhandled UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1001720003}>: COMPILE-FILE-ERROR while compiling #<PROTOBUF-SOURCE-FILE "cl-protobufs" "well-known-types" "descriptor">

cl-protobufs/tests fails to build because of a failure in cl-protobufs.

Full log here

It's not possible to build the CL plugin without this header file (among others), which is missing from the official repo, and can't be found anywhere, because it's a Google internal file (as it seems), and they don't want to expose it as public API:

<google/protobuf/stubs/strutil.h>

My question: is there any place on the interwebs to get the missing header files? I tried some workaround which compiles, but then it becomes a linking nightmare...

This is more of a feature request than an issue, but it's certainly important:

What about using cl-protobufs on e.g. a mobile device, where we don't have (and don't need) the 'protoc' executable installed, because the respective *.lisp files have already been generated.

In other words: what about just a passive use of a minimal version of cl-protobufs, without full installation?

Is this currently possible? (Looking at the JS protobufs implementation, they certainly seem to provide this.)

• Lispworks Version: 8.0.1 (64-bit) for macOS
• mac OS version: Ventura 13.0
• mac Silicon: Intel

When I attempt to load cl-protobufs using quicklisp, I get the following error:

CL-USER 5 > (ql:quickload :cl-protobufs)
To load "cl-protobufs":
  Load 1 ASDF system:
    cl-protobufs
; Loading "cl-protobufs"

**++++ Error between functions:
  Reader cannot find package LISPWORKS-FLOAT.

**++++ Error between functions:
  Reader cannot find package LISPWORKS-FLOAT.

**++++ Error between functions:
  Reader cannot find package LISPWORKS-FLOAT.

**++++ Error between functions:
  Reader cannot find package LISPWORKS-FLOAT.
; *** 4 errors detected, no fasl file produced.

Error: COMPILE-FILE-ERROR while compiling
#<ASDF/LISP-ACTION:CL-SOURCE-FILE "cl-protobufs" "models" "float-bits">
  1 (continue) Retry compiling
#<ASDF/LISP-ACTION:CL-SOURCE-FILE "cl-protobufs" "models" "float-bits">.
  2 Continue, treating compiling
#<ASDF/LISP-ACTION:CL-SOURCE-FILE "cl-protobufs" "models" "float-bits">
    as having been successful.
  3 Retry ASDF operation.
  4 Retry ASDF operation after resetting the configuration.
  5 Retry ASDF operation.
  6 Retry ASDF operation after resetting the configuration.
  7 (abort) Give up on "cl-protobufs"
  8 Register local projects and try again.
  9 Return to top loop level 0.

Any inputs to resolve this?

Just a note: Works fine on the same mac with SBCL 2.2.10

struct.proto is a well known type, and thus should be loaded by the well known types module.

The messages (and one enum!) have special JSON mappings which will also need to be implemented.

This shouldn't be done until #181 is fixed.

I tryed to build protoc-gen-cl-pb on mac m1 but got this error

❯ PROTOC_ROOT=/opt/homebrew/Cellar/protobuf/21.3 make g++ enum.o field.o file.o generator.o literals.o message.o names.o service.o proto2-descriptor-extensions.o main.o -o protoc-gen-cl-pb -Wl,-L/opt/homebrew/Cellar/protobuf/21.3/lib -Wl,-rpath -Wl,/opt/homebrew/Cellar/protobuf/21.3/lib -lprotoc -lprotobuf -pthread Undefined symbols for architecture arm64: "google::protobuf::internal::ExtensionSet::RegisterExtension(google::protobuf::MessageLite const*, int, unsigned char, bool, bool)", referenced from: void google::protobuf::internal::StringTypeTraits::Register<google::protobuf::FileOptions>(int, unsigned char, bool) in proto2-descriptor-extensions.o void google::protobuf::internal::StringTypeTraits::Register<google::protobuf::MessageOptions>(int, unsigned char, bool) in proto2-descriptor-extensions.o void google::protobuf::internal::StringTypeTraits::Register<google::protobuf::FieldOptions>(int, unsigned char, bool) in proto2-descriptor-extensions.o "google::protobuf::internal::ArenaStringPtr::Set(google::protobuf::internal::ArenaStringPtr::EmptyDefault, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::Arena*)", referenced from: google::protobuf::io::AnnotationProtoCollector<google::protobuf::GeneratedCodeInfo>::AddAnnotation(unsigned long, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<int, std::__1::allocator<int> > const&) in generator.o ld: symbol(s) not found for architecture arm64 clang: error: linker command failed with exit code 1 (use -v to see invocation) make: *** [Makefile:29: protoc-gen-cl-pb] Error 1

Consider the following proto:

syntax = "proto2"

message Foo {
  map<string,Bar> map_field = 1;
}

message Bar {
  optional int32 int_field = 1;
}

When the accessor foo.map-field-gethash is being generated, cl-protobufs calls (find-message bar) to check if bar constitutes a message type. This will return nil as the message has not yet been processed yet. So, the output type of foo.map-field-gethash gets set to bar.

This is a problem, since nil is the default for any message type, and nil is not of type bar. Hence, calling the accessor on any un-set key wil result in a type error.

The simple fix is to just set the underlying type of the hash-table to be (or null message) via protoc, which is what we do for all message fields. Sadly, this allows us to insert nil into the map, but we cannot get around this until there is a proper proto-kind slot on the field descriptor (or outputted by protoc).

Referring to the math example; I assume we would:

• Generate the math.proto.lisp file from the math.proto
• Then use the generated file in the example math.lisp

My question is: does one need to explicitly load the math.proto.lisp or not? How would math.lisp work without loading the generated lisp file?

When I try to load the generated lisp file, get a package error:

* (load "/tmp/math.proto.lisp") 

 The name "CL-PROTOBUFS.IMPLEMENTATION" does not designate any package.

We should do this for several reasons:

1. It's more exact, we're generating the common lisp proto files.
2. We have conflicts with Robert Browns protocol buffer implementation.

I have been looking at gRPC for IPC and it looks to fit the bill nicely. Except perhaps that there is no server for Common Lisp. I'm hoping that I just missed something here, or some library elsewhere.

It sort of looks like HTTP/2 is a stumbling block. I've started to revive cl-http2-protocol, but may be getting in over my head with these network protocols I've never worked with before.

Before I go too far down this path, I thought I'd ask if there a CL gRPC server anywhere? Bonus points for a macro/function that I can point at a package and have the functions within exposed automatically.

I'm having problems installing cl-protobufs. I get errors like

g++ -std=c++14 -I/PATH/TO/MY/LOCAL/lisp/protobuf/include -I.   -c -o generator.o generator.cc
generator.cc: In member function ‘virtual bool google::protobuf::cl_protobufs::LispGenerator::Generate(const google::protobuf::FileDescriptor*, const std::string&, google::protobuf::compiler::OutputDirectory*, std::string*) const’:
generator.cc:62:5: error: ‘GOOGLE_CHECK’ was not declared in this scope
   62 |     GOOGLE_CHECK(annotations.SerializeToZeroCopyStream(meta.get()));
      |     ^~~~~~~~~~~~
generator.cc: In member function ‘virtual bool google::protobuf::cl_protobufs::LispGenerator::GenerateAll(const std::vector<const google::protobuf::FileDescriptor*>&, const std::string&, google::protobuf::compiler::GeneratorContext*, std::string*) const’:
generator.cc:135:5: error: ‘GOOGLE_CHECK’ was not declared in this scope
  135 |     GOOGLE_CHECK(annotations.SerializeToZeroCopyStream(meta.get()));
      |     ^~~~~~~~~~~~
make: *** [<builtin>: generator.o] Error 1

FWIW, I had to use different compilation steps for protobuf than what's included in your README, since protobuf seems to be currently set up for bazel or cmake builds. After trying various approaches, the following are the steps that wound up successfully compiling protobuf and installing it (and its include/) to a local user directory for me. I don't get an error until I try to compile cl-protobuf:

export PROTOC_ROOT=/PATH/TO/MY/LOCAL/lisp/protobuf
mkdir -p /tmp/lisp/proto
pushd /tmp/lisp/proto

git clone --recursive https://github.com/google/protobuf
pushd protobuf
git submodule update --init --recursive

mkdir -p ${PROTOC_ROOT}
cmake . -DCMAKE_CXX_STANDARD=14 -DCMAKE_INSTALL_PREFIX=${PROTOC_ROOT}
cmake --build . --parallel 10

cmake --install .
path-prepend ${PROTOC_ROOT}/bin
popd # protobuf

git clone [email protected]:qitab/cl-protobufs.git

pushd cl-protobufs/protoc
make  #### ****  <-- error occurs here

popd # cl-protobufs/protoc
popd # /tmp/lisp/proto

Copy and pasted from internal bug b/159352737:

Setters, at least message and group setters, should check set type.
We set the type of on group/ message fields in proto to be
(or nil message-type) instead of message-type. This is so
we don't cons an unneeded proto structure.
This means we can setf the field to nil without, which shouldn't be allowed.
We can fix this by type-checking the input.

We should also declare the return type of an accessor.
This will allow a smarter compilation.

At the moment, I'm unable to compile the proto files for opentelemetry. I have an asd like so:

(defsystem "opentelemetry-cl"
  :version "0.1.0"
  :depends-on (:cl-protobufs)
  :defsystem-depends-on (:cl-protobufs.asdf)
  :components ((:module "opentelemetry-proto"
                :serial t
                :components
                ((:protobuf-source-file "common"
                  :proto-pathname "opentelemetry/proto/common/v1/common.proto"
                  :proto-search-path ("./"))
                 (:protobuf-source-file "resource"
                  :proto-pathname "opentelemetry/proto/resource/v1/resource.proto"
                  :depends-on ("common")
                  :proto-search-path ("./"))))))

opentelemetry-proto is cloned to a subdirectory, which can be done with this command:

$ git clone --branch=v1.1.0 https://github.com/open-telemetry/opentelemetry-proto opentelemetry-proto

I'm using qlot as well:

$ qlot add qitab/cl-protobufs
$ qlot install

When attempting to load the system, I receive an error the following error message:

$ qlot exec rlwrap ros run -s opentelemetry-cl

debugger invoked on a CL-PROTOBUFS:PROTOBUF-ERROR in thread
#<THREAD tid=1273506 "main thread" RUNNING {10013E0003}>:
  Could not find file "opentelemetry/proto/common/v1/common.proto" imported by #<CL-PROTOBUFS:FILE-DESCRIPTOR RESOURCE (package opente
lemetry.proto.resource.v1) {1004AF6673}>

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [RETRY                        ] Retry compiling
   #<PROTOBUF-SOURCE-FILE "opentelemetry-cl" "opentelemetry-proto" "resource">.
  1: [ACCEPT                       ] Continue, treating compiling
   #<PROTOBUF-SOURCE-FILE "opentelemetry-cl" "opentelemetry-proto" "resource">
                                     as having been successful.
  2:                                 Retry ASDF operation.
  3: [CLEAR-CONFIGURATION-AND-RETRY] Retry ASDF operation after resetting the
                                     configuration.
  4:                                 Retry ASDF operation.
  5:                                 Retry ASDF operation after resetting the
                                     configuration.
  6: [ABORT                        ] Give up on "opentelemetry-cl"
  7: [REGISTER-LOCAL-PROJECTS      ] Register local projects and try again.
  8: [CONTINUE                     ] Ignore runtime option --eval "(ros:run '((:eval\"(ros:quicklisp)\")(:system \"opentelemetry-cl\")))".
  9:                                 Skip rest of --eval and --load options.
 10:                                 Skip to toplevel READ/EVAL/PRINT loop.
 11: [EXIT                         ] Exit SBCL (calling #'EXIT, killing the process).

(CL-PROTOBUFS:PROTOBUF-ERROR "Could not find file ~S imported by ~S" "opentelemetry/proto/common/v1/common.proto" #<CL-PROTOBUFS:FILE-DESCRIPTOR RESOURCE (package opentelemetry.proto.resource.v1) {1004AF6673}>)
   source: (ERROR 'PROTOBUF-ERROR :FORMAT-CONTROL FORMAT-CONTROL
            :FORMAT-ARGUMENTS FORMAT-ARGUMENTS)

It's saying that it can't find the proto file for common.proto, but as far as I can tell, the search path is setup correctly. It's failing in the validate-imports function, but it's not exactly clear to me what I need to do to get common.proto to resolve in *file-descriptors*.

A few things I've noticed:

• removing the resource component seems to have successful compilation (as in, only compiling common)
• running protoc to create generate the lisp files seems to work without issues:

$ protoc \
  --proto_path=$PWD/opentelemetry-proto/ \
  --cl-pb_out=output-file=resource.lisp:$PWD \
  $PWD/opentelemetry-proto/opentelemetry/proto/resource/v1/resource.proto
$ ls -lah resource.lisp
-rw-r--r-- 1 mike 1.5K Feb 28 19:21 resource.lisp

• I've attempted multiple different ways to declare the pathnames for the module/components, to no clear solution

Is there anything obvious that I may be doing wrong?

Thanks in advance.

Currently, has-* functions are not created for any optional proto3 field.

However, the C++ protobuf library creates has-* functions for proto3 message fields.

We can already simulate this, since a message field is unset if and only if the slot is bound to nil, however this isn't reflected in the has-* function, as we currently have no way to distinguish between enum and message types during the code generation step.

It's crazy we rebuild it on every submit.
Shouldn't we be able to cache?

Hi.

This is more a question as I couldn't find the answer.

I've tried a few variants:

protoc --plugin=protoc-gen-lisp=/usr/local/bin/protoc-gen-lisp \
  --lisp_out="`pwd`/folder/lisp" \
  --proto_path=folder \
  $(find folder -iname "*.proto")

or

protoc --plugin=protoc-gen-lisp=/usr/local/bin/protoc-gen-lisp \
  --lisp_out="`pwd`/folder/lisp" \
  --proto_path=folder \
  folder/*.proto

The cl plugin then responds with:

--lisp_out: protoc-gen-lisp: First file chunk returned by plugin did not specify a file name.

So, I'm not sure if the --lisp_out form is correct as in the documentation it uses output-file, but here it's multiple files.

Any advice?

Manfred

Building with SBCL 2.3.1.67-d262a01cc / ASDF 3.3.5 for quicklisp dist creation.

Trying to build commit id 2199e5f

cl-protobufs fails to build with the following error:

; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE FIELD-DESCRIPTOR-PROTO ...))
;   The value TYPE-DOUBLE is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE FILE-OPTIONS ...))
;   The value SPEED is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE FIELD-OPTIONS ...))
;   The value STRING is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
; caught COMMON-LISP:ERROR:
;   (during macroexpansion of (PI:DEFINE-MESSAGE METHOD-OPTIONS ...))
;   The value IDEMPOTENCY-UNKNOWN is not of type KEYWORD when setting slot PI::NAME of structure CL-PROTOBUFS:ENUM-VALUE-DESCRIPTOR
...
Unhandled UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1001718003}>: COMPILE-FILE-ERROR while compiling #<PROTOBUF-SOURCE-FILE "cl-protobufs" "well-known-types" "descriptor">

cl-protobufs/tests fails to build because of a failure in cl-protobufs.

Full log here

this was found in #212

Building with SBCL 2.0.9.5-442f54894 / ASDF 3.3.1 for quicklisp dist creation.

Commit id 4d9d282

cl-protobufs fails to build with the following error:

; caught COMMON-LISP:ERROR:
;   COMMON-LISP:READ error during COMMON-LISP:COMPILE-FILE: Package COM.GOOGLE.BASE does not exist. Line: 12, Column: 48, File-Position: 343 Stream: #<SB-INT:FORM-TRACKING-STREAM for "file /home/quicklisp/.cache/common-lisp/sbcl-2.0.9.5-442f54894-linux-x64/home/quicklisp/quicklisp-controller/dist/build-cache/cl-protobufs/aa82216c6f9a999fdc71ede466c4eb6f42917ef0/cl-protobufs-20201006-git/any.lisp" {10072473C3}>
...
Unhandled UIOP/LISP-BUILD:COMPILE-FILE-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {1001A50103}>: COMPILE-FILE-ERROR while compiling #<PROTOBUF-SOURCE-FILE "cl-protobufs" "well-known-types" "any">

cl-protobufs/tests fails to build because of a failure in cl-protobufs.

Full log here

enum FSBossStrongholdEvent {
  ready = 0;
  start = 1;
  finish = 2;
}

(pi:define-enum fs-boss-stronghold-event
    (:name "FSBossStrongholdEvent")
  (ready :index 0)
  (start :index 1)
  (finish :index 2))

 enum SAStage {
  READY = 0; 
  SIGN_UP = 1; 
  GROUP = 2; 
  TOP32 = 3; 
  FINAL = 4; 
  FINISH = 5; 
  CANCEL = 6; 
}

(pi:define-enum sa-stage
    (:name "SAStage")
  (ready :index 0)
  (sign-up :index 1)
  (group :index 2)
  (top32 :index 3)
  (final :index 4)
  (finish :index 5)
  (cancel :index 6))

fs-boss-stronghold-event`s finish conflict with sa-stags`s finish

and got this error

The constant +FINISH+ is being redefined (from 5 to 2)
   [Condition of type SB-EXT:DEFCONSTANT-UNEQL]