embedded-ams / embeddedproto

Embedded Proto is a C++ Protocol Buffers implementation specifically suitable for microcontrollers. It is small, reliable and easy to use.

Embedded Proto is a product of Embedded AMS B.V. For more information about Embedded Proto please visit EmbeddedProto.com.

Copyrights 2020-2024 Embedded AMS B.V., www.EmbeddedAMS.nl, [email protected]

Introduction

Embedded Proto is a C++ implementation of Google Protocol Buffers dedicated to microcontrollers. The implementation focuses on a small footprint and low memory usage. No dynamic memory allocation is used to make the code predictable. Unit tests and static code analysis are used to improve the reliability of the code.

Natively, protocol buffers are not suitable for microcontrollers. The C++ generated is written for server and desktop processors. This is where Embedded offers a solution. Embedded Proto is a plugin for protoc generating C++ code suitable for microcontrollers. In this way, Embedded Proto provides an easy-to-use interface to exchange data between embedded devices and the outside world. Specify the data format between your IoT device and other devices, servers, apps, or desktop applications in a standardized way!

This document details the following:

• What is new
• License
• Installation
• Usage
• Supported Features
• Examples
• Development

What is new

To stay up to date, signup for our User Update.

3.5.0

• Bub fix related to optional string or bytes fields. They where not check to be set or not.
• Small interface change in RepeatedField class to use correct array index type. Tjos is a possible breaking change when you derived from the RepeatedField.
• Support for spaces in folder names on Windows.
• Started reworking the company internal toolchain.

3.4.0

• In the background the installation switched to using Python SetupTools (thanks to the contributors). In the future we would like to use Pip for the installation.
• Problems with non-matching versions have been addressed. You now get a warning which allows you to continue even if the version does not match exactly.
• Added some useful command line options to the setup script.

3.3.0

• Added a to_string function for debugging (see documentation).
• Added getter functions which will return an error for index out of bounds cases.
• Bug fix the toposorting algo for nested message definitions.

3.2.0

The most notable improvements in this version are:

• Updated to protobuf v21.5. The python module made by Google for this version is not backwards compatible. Please update your protoc installation!
• Wrote a python setup script instead of separate scripts for Linux and Windows.
• Added simple implementations of the ReadBufferInterface and WriteBufferInterface: ReadBufferFixedSize and WriteBufferFixedSize.

3.1.0

The most notable improvements in this version are:

• Reworked code to extend the support back to C++11.
• Worked on optimizing running the code coverage in Sonarqube.

3.0.0

The most notable improvements in this version are:

• The length of repeated, string and bytes fields can now be set from the .proto file. You can find information on how to do this in the online documentation.
• The ram size of messages has been reduced. This was done by using less polymorphism in the low-level field classes. This required upgrading to C++17 and up.
• In a .proto file, it is now possible to use a message or enum before it is defined. The plugin will make a dependency tree of the messages and enums defined and sort them before generating the source code. Recursive inclusion is not supported.
• Some of the message functions changed. The functions were already marked as deprecated in the latest 2.X.X release.

License

Embedded Proto uses a dual licensing model. One for open source projects and one for commercial usage.

Open Source

You can use Embedded Proto for free in open source projects or for testing. However, on demand support is not available only if you have a commercial license. For open source projects, you can download the source code from Github. The code is licensed under the GNU General Public License V3.0, which you can use for all your non-commercial projects.

Commercial License

Are you developing a commercial product? If so, you need to buy a commercial license from Embedded Proto. There is a suitable license for each type of business, from startup to enterprise. Depending on the license, it may give you access to the following:

• An unlimited number of mcu’s
• Professional support
• Code quality report

You can request more information about a commercial license on our website.

Installation

What is required to be able to generate source files based on .proto files:

1. Python 3.8 and up
2. Pip
3. Protobuf v21.5 and up (tested with v25.1)
4. Git

After installing the requirements, continue by cloning the Embedded Proto repo. We advised using Embedded Proto as a submodule in your project. This way, you can track the version of Embedded Proto with the version of your project.

cd your_project_dir
git submodule add https://github.com/Embedded-AMS/EmbeddedProto.git
git commit -m "Added the latest version of Embedded Proto as a submodule."

Next, enter the Embedded Proto folder and run the setup script. The script will create a self-contained python environment. In this environment, various python packages will be installed, which are required by Embedded Proto.

cd EmbeddedProto
python setup.py

Did you install protoc in a custom folder, or is the include folder of protobuf not in your path? In these cases, you may get an error from the setup script. You have to provide the location of the include with the --include parameter:

python setup.py --include ~/protobuf/protoc-21.5/include

In this example, you have installed a specific version of protoc, and you named its installation folder ~/protobuf/protoc-21.5.

You can check out latest the command line parameters of the setup script using the help parameter:

python setup.py --help

More installation documentation can be found on the documentation website.

Usage

You write your proto files defining the message structure when working on your project. Next, you would like to use them in your source code. Generating the code based on your message definitions is required. Please do this by using our plugin for the protoc compiler protoc-gen-eams.py. Generate the code using the following command:

On Linux:

protoc --plugin=protoc-gen-eams -I./LOCATION/PROTO/FILES --eams_out=./generated_src PROTO_MESSAGE_FILE.proto

On Windows:

protoc --plugin=protoc-gen-eams=protoc-gen-eams.bat -I.\LOCATION\PROTO\FILES --eams_out=.\generated_src PROTO_MESSAGE_FILE.proto

Protoc is instructed to use our plugin with the option --plugin. The standard option -I includes the folder where your *.proto files are located. The option --eams_out specifies where to store the generated source code. Finally, the protofile to be parsed is specified.

As our plugin is a Python script and the protoc plugin should be an executable, a small terminal script is included. This terminal script is called protoc-gen-eams and is used to execute python with the Embedded Proto python script as a parameter. The main takeaway is that this script should be accessible when running your protoc command.

After running protoc without errors, the generated source code is located in the folder specified by -eams_out. You have to include two folders in your toolchain:

• The folder you specified with -eams_out, and
• The source code of Embedded Proto is located in EmbeddedProto/src.

Examples

Our website hosts an array of examples detailing possible use cases and tutorials on toolchain integrations. This includes:

• A Command and Control structure,
• Ethernet Communication,
• and Makefile integration.

Supported Features

Below two tables indicate the level of support for various variable types and features.

Variable Type	Support
double	Full
float	Full
int32	Full
int64	Full
uint32	Full
uint64	Full
sint32	Full
sint64	Full
fixed32	Full
fixed64	Full
sfixed32	Full
sfixed64	Full
bool	Full
string	Length fixed via template or custom option
bytes	Length fixed via template or custom option

Feature	Support
Enum	Full
Messages as variables	Full
Defining messages in messages	Minimal
oneof	Full
singular	Full
repeated	Length fixed via template or custom option
optional	Full

All features mentioned above are of version proto3. At this moment, proto2 is not supported. Taken from the Protobuf website:

Prefer proto3 while proto2 will continue to be supported, we encourage new codes to use proto3 instead, which is easier to use and supports more languages. For this reason, it is unlikely that Embedded Proto will support proto2 in the future.

Development

If you consider helping with the development of Embedded Proto please consider reading this. It details how you can build the unit tests included in this repo.

embeddedproto's Issues

Oneof "selection" is not serialized when a field in the oneof is set to its default value

When a field of a oneof block is set to its default value, the information which field in the oneof was set is lost when the message is serialized.
(This case is explixitly mentioned in the Documentation: "If you set a oneof field to the default value (such as setting an int32 oneof field to 0), the 'case' of that oneof field will be set, and the value will be serialized on the wire.")
This simple example demonstrates the issue:

message Foo {
  oneof bar{
      int32 value1 = 1;
      int32 value2 = 2;
  }
}

When using this message and setting one of the two value-variables to 0 (which is their default value), it is not serialized and it can not be determined which value was set on the original message after deserializing the message.

Foo testMessage;
testMessage.set_value1(0); // set a variable in the oneof to its default value

// readout which of the values is set: VALUE1
auto which = testMessage.get_which_bar();

// serialize
WriteBufferImpl writeBuffer;
testMessage.serialize(writeBuffer);
// deserialize
ReadBufferImpl readBuffer(writeBuffer.get_buffer(), writeBuffer.get_size());
Foo testMessageDeserialized;
testMessageDeserialized.deserialize(readBuffer);

// readout which value in the oneof is set -> fails: NOT_SET
auto which_deserialized = testMessageDeserialized.get_which_bar();

In the generated code, there is a check if the value is equal to 0 (the first condition in the if statement). If the value is inside a oneof field, this check should not be performed because when the corresponding value is selected (which_bar_ equals VALUE1) the value should always be serialized, even if it is 0.

[...]
switch(which_bar_)
{
  case id::VALUE1:
    if((0 != bar_.value1_.get()) && (::EmbeddedProto::Error::NO_ERRORS == return_value))
    {
      return_value = bar_.value1_.serialize_with_id(static_cast<uint32_t>(id::VALUE1), buffer, false);
    }
    break;
[...]

Best regards
Michael

Generated code not compiling with gcc-arm-none-eabi-10-2020-q2-preview

Different issues on two different toolchains:
ARM GCC 10: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads
Linux GCC 10.1, installed from Ubuntu's gcc-10 package.

python 3.6.9

With ARM GCC 10, I'm getting errors like this:

EmbeddedProto/src/RepeatedField.h:54:34: error: expected unqualified-id before '=' token
   54 |     static constexpr bool PACKED = !std::is_base_of<MessageInterface, DATA_TYPE>::value;

EmbeddedProto/src/RepeatedField.h:137:12: error: expected ')' before '__attribute__'
  137 |         if(PACKED)
      |           ~^
      |            )

Maybe there are some issues with attributes with this toolchain? I'm not sure.

With Linux GCC 10.1, I'm getting errors like this:

proto/example.h:54:16: error: 'B' does not name a type
   54 |   inline const B& b() const { return b_; }

This is because the generated code generates message A first, without forward declaring message B.

Proto file:

syntax = "proto3";

message A {
  B b = 1;
}

message B {
  int32 val = 1;
}

question on the out-of-bound access of a FieldBytes

I have a question regarding how this library defines out-of-bound access on a FieldBytes.

Scenario 1

EmbeddedProto/test/test_string_bytes.cpp

Line 281 in 3fe7922

EXPECT_EQ(0, msg.get_b().get_const(0));

If msg was cleared, the length is set to zero, then accessing msg.get_const(0) should be considered invalid. However, in the unittest, the element is accessed and zero is expected.
This library seems not using C++ exception but I think this situation should be asserted rather than defined (to return 0) to prevent potential bugs.

Scenario 2

EmbeddedProto/test/test_string_bytes.cpp

Line 294 in 3fe7922

EXPECT_EQ(11, msg.get_b().get_const(10));

When the MAX_LENGTH is exceeded, the index will point to the last element, according to the unittest. However, the intended message is tampered so this mechanism (refer to the last if out-of-bound) seems not making sense to me. Is there a specific reason why it is designed in this way?

Thank you

What's your opinion?

Can't install EmbeddeProto

I use windows 10 Pro, 64bit.
I use ESP_IDF for ESp32, so Python, Git...I have it installed from before.
By opening CMD, and the folder "cd EmbeddedProto", calling command "python setup.py" or "python setup.py --include ~/protobuf/protoc-21.5/include" it generates me an error: (see Picture)

Please help.

Macro collision with PACKED in nrf52 SDK

I am using the nrf52 SDK, and unfortunately, they define a PACKED macro deep in their library, which is causing a collision with RepeatedField.h's static constexpr bool PACKED.
I realize this isn't a problem with your library, and I can also change it locally in my copy of EmbeddedProto. But, I would like if EmbeddedProto could choose a different name for this variable.

Maybe consider Google's constants naming convention, which would make this kPacked? https://google.github.io/styleguide/cppguide.html#Constant_Names

Setup script fails with protoc minor updates

The setup script seems to fail comparing protoc versions with required versions.

We attempt a regular expression parse ..

However the script fails comparing minor updates and it is now on the user to ensure compatibility

Original script fails with:

The version of Protoc (v0.None) you have installed is not compatible with the version of
the protobuf python package (v21.6) Embedded Proto requires. These are your options:
	1. Install a matching version of Protoc.
	2. Change the version of Embedded Proto.

First of all inherently minor updates to API's should not be considered API breaking, I find it odd that we have to version control our protoc to match the EmbeddedProto tool without any leniency.

Also protoc --versions apperars to only output 24.0 .

User is required to update the available protobuf versions and ensure compatibility with the read protoc version.

FIX: User can update the script themselves and approve minor versions up to their installed version. In this case '24.0'

---

Solution:

Issue warning only for minor version incompatibility updates to protobuf. It seems odd to restrict the protoc minor version updates, perhaps there needs to be a CI process in place that is subordinate to the protobuf schedule as well.

Request - template based fixed size string/bytes becomes a bit repetitive/bugprone

Before describing this feature request I want to express my gratitude for this awesome work! It's been an absolute pleasure to use this for my thesis project for making an UART CLI, LoRa communication interface and flash/SD storage interface🥇 🥇 🥇

Background
From my understanding Google's Protobuf does not have the concept of fixed-size fields, which seem power this library (and with good reason!)

Problem description
Having multiple fixed-size fields causes the template variables to become a bit confusing. Especially when using certain message types across multiple message processor functions, I started to repeat the same template variables over and over again to eventually come up with a define to replace this with a define:

#define PROTO_LIMITS MAX_UART_FIELD1_BYTES, MAX_UART_FIELD2_BYTES, MAX_UART_FIELD3_BYTES etc etc
UartCommand<PROTO_LIMITS > uartCommand;
// Or even
#define UART_COMMAND UartCommand<PROTO_LIMITS>

This occurs mostly for nested fields which have nothing to do with each other. For example with OneOf I've started to notice that I had to add the same template variable multiple times - a bugprone process if you dont oversee the fields in use.

message UartCommand { 
    oneof Body { 
        RadioRxConfig rxConfig = 1; 
        RadioTxConfig txConfig = 2; 
        DeviceConfiguration deviceConfiguration = 3;
        RequestBootInfo requestBootInfo = 4; 
        ClearMeasurementsCommand clearMeasurementsCommand = 5;
        LoRaMessage transmitCommand = 6;
        LoRaReset resetRadio = 7;
    } 

    bool doNotProxyCommand = 8;
    // ...
} 

Discussion
I would like to discuss the options. Can we add or think up anything which allows us to fixate the field size for all usages? Maybe an optional define which gets consumed by precompilation instead of the template argument?

Im fully aware that fixated defines contradict the decoupling which is achieved by template arguments. I've found that across my application Im mostly hard-coupling the field constraints.

I guess you must have thought a lot about this by now, so Im open to hear your thoughts! 👍🏼

Support for DebugString()

I noticed that the header file for the embedded output does not provide DebugString() to provide a human-readable form for debugging purposes. Do you have any recommendations on how this could be provided, whether automatically generated by your protoc plugin, or manually on an as-needed basis? Thanks!

Support repeated enum fields

Hello,

We noticed that repeated enum fields are currently not supported

syntax = "proto3";
package test;
import "embedded_proto/options.proto";

enum Config {
  CONFIG_UNSPECIFIED = 0;
  CONFIG_UP = 1;
  CONFIG_DOWN = 2;
}

message PayloadEnvelope {
  repeated Config config = 1 [ (embedded_proto.options).maxLength = 4];
}

crashes with

In file included from ./RepeatedFieldFixedSize.h:34,
from ./test.h:30,
./RepeatedField.h: In instantiation of 'class EmbeddedProto::RepeatedField<test::Config>':
./RepeatedFieldFixedSize.h:52:9: required from 'class EmbeddedProto::RepeatedFieldFixedSize<test::Config, 4>'
required from here
./RepeatedField.h:93:74: error: static assertion failed: A Field can only be used as template paramter.
93 | static_assert(std::is_base_of_v<::EmbeddedProto::Field, DATA_TYPE> || IS_BASIC_TYPE,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~

The only workaround is

syntax = "proto3";
package test;
import "embedded_proto/options.proto";

message Config
  enum ConfigType {
    CONFIG_UNSPECIFIED = 0;
    CONFIG_UP = 1;
    CONFIG_DOWN = 2;
  }
  ConfigType config = 1;
}

message PayloadEnvelope {
  repeated Config config = 1 [ (embedded_proto.options).maxLength = 4];
}

which costs 1-2 bytes for each repeated element additionally. Hence, direct support of repeated enum fields would save quite a few bytes in the serialized byte stream

Best,
ChiSaw

setup.sh uses pip2 when both pip2 and pip3 are available

Since EmbeddedProto uses python3, the setup script should also use pip3 explicitly, instead of relying on pip to point to pip3.

The effect this had was after running setup.sh, when I ran protoc, protoc-gen-eams couldn't import jinja2, because it had been installed in the python2 environment.

Decoding/skipping unknown non-empty length-delimited fields

It seems that during decoding, if there is an unknown length-delimited field that is not empty, decoding the protobuf will fail with ERROR:INVALID_WIRETYPE. This does not happen if the unknown field is empty (with zero length) or has varint wire type.

If I add the unknown field to my .proto file as a string field and account for the capacity of the decoded C-string with an additional template argument to the generated class template, the problem goes away.

Is this expected/intended? In the target use case there are fields that I don't care about and do not need decoding but maybe the code generated by EmbeddedProto is insisting on the decoding of unknown length-delimited fields, including but not limited to string fields.

mutable accessor doesn't seems to work as expected

Hello, first of all, thank you for this amazing library!

I try to use the mutable accessors but it doesn't seem to do what the name suggest.

Here is the relevant part of my .proto file:

message BatteryReport {
    float voltage = 1;
}

message UpMessage {
    oneof inner{
        OdomReport odom_report = 1;
        BatteryReport battery_report = 2;
    }
}

And here is the code I tried. I believe it should work but it doesn't. Is it a normal behavior?

// Does not work!
UpMessage msg;
auto battery_report = msg.mutable_battery_report();
battery_report.set_voltage(voltage);
msg.serialize(*buffer);
// the "voltage" field is still set to its default value.

Those other code works well:

// OK
UpMessage msg;
BatteryReport battery_report;
battery_report.set_voltage(voltage);
msg.set_battery_report(battery_report);
msg.serialize(*buffer);


// Ok
UpMessage msg;
auto battery_report = msg.battery_report();
battery_report.set_voltage(voltage);
msg.set_battery_report(battery_report);
msg.serialize(*buffer);


// Ok
UpMessage msg;
auto battery_report = msg.mutable_battery_report();
battery_report.set_voltage(voltage);
msg.set_battery_report(battery_report);
msg.serialize(*buffer);

Windows installation instructions not working

When following the installation instructions I get the following output in Powershell:

`EmbeddedProto> .\setup.bat C:\ProgramData\chocolatey\lib\protoc\tools\include\google\protobuf

Collecting Jinja2==3.0.3
Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)

Collecting MarkupSafe==2.0.1
Using cached MarkupSafe-2.0.1-cp310-cp310-win_amd64.whl (15 kB)

ERROR: Could not find a version that satisfies the requirement pkg_resources==0.0.0 (from versions: none)

ERROR: No matching distribution found for pkg_resources==0.0.0

'.\venv\Scripts\deactivate.bat)' is not recognized as an internal or external command,
operable program or batch file.`

String fields use an undefined setter method.

Hello,

I've found a potential issue with the auto-generated header code when it comes to message fields of protobuf type string (which becomes type ::EmbeddedProto::FieldString<_LENGTH>).

To start, I am seeing this on tag 2.1.0 and on the latest (at the time of this writing) master.

Here is the issue I'm seeing. Let's say you define a protobuf message like so:

message FooMessage {
  string foo_string = 1;
  uint64 foo_int = 2;
}

In the generated header a portion of the class FooMessage will look like this:

template<uint32_t foo_string_LENGTH>
class FooMessage final: public ::EmbeddedProto::MessageInterface
{
  public:
    FooMessage() :
        foo_string_(),
        foo_int_(0U)
    {

    };
    ~FooMessage() override = default;

    FooMessage& operator=(const FooMessage& rhs)
    {
      set_foo_string(rhs.get_foo_string());
      set_foo_int(rhs.get_foo_int());
      return *this;
    }

    inline void clear_foo_string() { foo_string_.clear(); }
    inline ::EmbeddedProto::FieldString<foo_string_LENGTH>& mutable_foo_string() { return foo_string_; }
    inline const ::EmbeddedProto::FieldString<foo_string_LENGTH>& foo_string() const { return foo_string_; }
    inline const char* get_foo_string() const { return foo_string_.get_const(); }

    inline void clear_foo_int() { foo_int_.clear(); }
    inline void set_foo_int(const EmbeddedProto::uint64& value) { foo_int_ = value; }
    inline void set_foo_int(const EmbeddedProto::uint64&& value) { foo_int_ = value; }
    inline EmbeddedProto::uint64& mutable_foo_int() { return foo_int_; }
    inline const EmbeddedProto::uint64& get_foo_int() const { return foo_int_; }
    inline EmbeddedProto::uint64::FIELD_TYPE foo_int() const { return foo_int_.get(); }

...

The issue is in the overload of the assignment operator. The method set_foo_string() is not defined and if this operator is used, a compilation error is generated.

    FooMessage& operator=(const FooMessage& rhs)
    {
      set_foo_string(rhs.get_foo_string());  // <----- THIS IS NOT DEFINED ANYWHERE
      set_foo_int(rhs.get_foo_int());
      return *this;
    }

After looking at the implementation of EmbeddedProto::FieldString, I would propose the following fix (which appears to compile):

    FooMessage& operator=(const FooMessage& rhs)
    {
      foo_string_ = rhs.get_foo_string();  // <----- CHANGE TO THIS
      set_foo_int(rhs.get_foo_int());
      return *this;
    }

This appears to do the right thing, but I might also be missing something. 🤷🏽‍♂️

Thank you for taking the time to read this and even more thanks for creating EmbeddedProto!

-a

Failure when importing descriptor.proto

I am adding a custom option via extending google.protobuf.FieldOptions and am getting the following error:

--eams_out: Embedded Proto error - Messages with repeated, string or byte fields use template parameters to define their length.For some reason it was not to add all required template parameters.

Here is a minimal reproducible case:

syntax = "proto3";

import "google/protobuf/descriptor.proto";

enum my_enum {
   ENUM_A = 0;
   ENUM_B = 1;
}

extend google.protobuf.FieldOptions {
   optional my_enum my_enum_option = 50000;
}

Adding import "google/protobuf/descriptor.proto"; on its own is enough to cause the failure.

I do not need the option available in EmbeddedProto generated code, but I do need it in other generated languages. I just need EmbeddedProto to not fail generation on this input.

empty message doesn't work

If you have an empty message like:

message A { }

...protoc generates this constructor:

  public:
    A() :
    {

    };

which has the ":" for an initializer list with nothing after it.

deserialize() does not consume payload bytes of unknown fields

When a field id is encountered that is not defined in the .proto file, deserialize() does not consume the payload bytes that follow and instead treats the first payload byte as the start of the next field and continues processing of the fields.

deserialize() should skip the payload bytes before trying to process the next field.

plugin is not working

Tried to review, if it can be used for cpp project. Running it caused error:

--eams_out: Embedded Proto error - list.remove(x): x not in list

template parameters not generated when including message in a different file, same package

I have 2 files, example.proto and example2.proto

example.proto:

package a.b;
import "path/to/example2.proto";
message X {
  X2 val = 1;
}

example2.proto:

package a.b;
message X2 {
  repeated int32 list = 1;
}

class X's val return value doesn't have the template parameters for list's length.

If I remove package a.b; from both files, it works correctly.

ImportError embedded_proto_options_pb2

Hello,
I updated EmbeddedProto from 3.3.3 to 3.4.2 with the latest protobuf version (25.2), and it crashes complaining about not finding embedded_proto_options_pb2.

I managed to resolve the issue by generating the file with:
protoc -I generator --python_out=generator/EmbeddedProto embedded_proto_options.proto

However this is not written in the documentation. Is it necessary? Could it be done in the setup.py so we don't have to manually generate it?

I did not changed my .proto file.

EDIT:
Curiously, I installed it on an other computer an it worked out of the box.
I will try investigate a bit more.

RepeatedField.h won't compile in ARM6 compiler

The ARM6 compiler is not happy with this struct definition and static_assert in RepeatedField.h:

//! Definition of a trait to check if DATA_TYPE is or is not a specialization of the FieldTemplate.
template<typename T>
static constexpr auto is_specialization_of_FieldTemplate_v = is_specialization_of_FieldTemplate<T>::value;

//! This class only supports Field and FieldTemplate classes as template parameter.
static_assert(std::is_base_of_v<::EmbeddedProto::Field, DATA_TYPE> || is_specialization_of_FieldTemplate_v<DATA_TYPE>, 
              "A Field can only be used as template parameter.");

It does not like having to cast is_specialization_of_FieldTemplate_v from auto to bool in the static_assert.
I fixed this by changing the type of is_specialization_of_FieldTemplate_v to bool from auto.

I'm not super confident in what exactly is going on in this code so I didn't want to make a PR but that is how I fixed it.

(So my new definition is:
//! Definition of a trait to check if DATA_TYPE is or is not a specialization of the FieldTemplate.
template
static constexpr bool is_specialization_of_FieldTemplate_v = is_specialization_of_FieldTemplate::value;
)

generated header guards missing path to file

The header guard still isn't correct, in a different (better...) way than #6

With a proto file at a/b/c.h, I expected header guards to include the path. Now, only the filename is used.

Expected: A_B_C_H
Got: C_H

This means if you have two proto files in different directories with the same name, the header guards will collide.

Linker Errors When Compiling For ESP32

Hi I recently tried using your library and I was able to get past the compiling stage.

however now i'm getting linker errors for WriteBufferFixedSize.h am I missing something?

or how exactly are you supposed to write to the buffer? any examples would be greatly appreciated thanks.

the functions that seem to be undefined are below

• EmbeddedProto::MessageInterface::serialize_with_id
• EmbeddedProto::MessageInterface::deserialize_check_type
• And WriteBufferFixedSize saying that the functions are not defined

on pretty much all the generated messages

Optional fields support

Hello!

Please, add optional fields support in proto3 syntax. This implemented by --experimental_allow_proto3_optional protoc flag. This feature will be added to protoc soon, but now it is experimental.

Error string: Api.proto: is a proto3 file that contains optional fields, but code generator protoc-gen-eams hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.

Cannot have a member named `id`

Hello, great project thanks for sharing!

I have a proto file that contains a member id:

message block_topology {
   bytes id = 1;
   uint64 height = 2;
   bytes previous = 3;
}

The generated code uses an enum id and causing a conflict while compiling.

Related error:

In file included from /Users/sgerbino/Projects/koinos-system-contracts/contracts/echo/echo.cpp:1:
/Users/sgerbino/Projects/koinos-system-contracts/contracts/echo/common.h:142:7: error: must use 'enum' tag to refer to type 'id' in this scope
      id id_tag = id::NOT_SET;
      ^
      enum
/Users/sgerbino/Projects/koinos-system-contracts/contracts/echo/common.h:99:27: note: enum 'id' is hidden by a non-type declaration of 'id' here
    inline const uint8_t* id() const { return id_.get_const(); }
                          ^
/Users/sgerbino/Projects/koinos-system-contracts/contracts/echo/common.h:147:30: error: must use 'enum' tag to refer to type 'id' in this scope
        id_tag = static_cast<id>(id_number);
                             ^
                             enum
/Users/sgerbino/Projects/koinos-system-contracts/contracts/echo/common.h:99:27: note: enum 'id' is hidden by a non-type declaration of 'id' here
    inline const uint8_t* id() const { return id_.get_const(); }

Test suite does not compile on Mac with clang

The test suite does not compile on Mac 10.14.6 with clang.

Steps to reproduce:

git clone --recursive https://github.com/Embedded-AMS/EmbeddedProto
cd EmbeddedProto
python3.7 -m venv venv
. venv/bin/activate
pip install -r requirements.txt
./build_test.sh

Enviroment:

• Mac OSX 10.14.6
• Shell: zsh or bash
• Compiler:

$ c++ -v
Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Target: x86_64-apple-darwin18.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

First errors (there are many more):

[ 86%] Building CXX object CMakeFiles/test_EmbeddedProto.dir/test/test_string_bytes.o
[ 91%] Building CXX object CMakeFiles/test_EmbeddedProto.dir/test/test_oneof_fields.o
In file included from /var/folders/vd/wr3sq1gn4vl6z129hcs5_kz00000gn/T/tmp.p8ASoSEH/EmbeddedProto/src/Fields.cpp:31:
In file included from /var/folders/vd/wr3sq1gn4vl6z129hcs5_kz00000gn/T/tmp.p8ASoSEH/EmbeddedProto/src/Fields.h:35:
/var/folders/vd/wr3sq1gn4vl6z129hcs5_kz00000gn/T/tmp.p8ASoSEH/EmbeddedProto/src/WireFormatter.h:259:19: error: function 'ZigZagDecode<unsigned long long>' with deduced return type cannot be used before it
      is defined
          value = ZigZagDecode(uint_value);
                  ^
/var/folders/vd/wr3sq1gn4vl6z129hcs5_kz00000gn/T/tmp.p8ASoSEH/EmbeddedProto/src/WireFormatter.h:482:29: note: 'ZigZagDecode<unsigned long long>' declared here
      static constexpr auto ZigZagDecode(const UINT_TYPE n) 
                            ^
/var/folders/vd/wr3sq1gn4vl6z129hcs5_kz00000gn/T/tmp.p8ASoSEH/EmbeddedProto/src/WireFormatter.h:416:75: error: no member named 'ceil' in namespace 'std'; did you mean simply 'ceil'?
        static constexpr uint8_t N_BYTES_IN_VARINT = static_cast<uint8_t>(std::ceil(
                                                                          ^~~~~~~~~
                                                                          ceil
/Library/Developer/CommandLineTools/usr/include/c++/v1/math.h:841:1: note: 'ceil' declared here
ceil(_A1 __lcpp_x) _NOEXCEPT {return ::ceil((double)__lcpp_x);}
^

Full log attached.
EmbeddedProto-mac-compilation-error-log.txt

header guards generated with "/" instead of "_", not ISO C++11 compliant

GCC complains:
example.h:33:9: warning: ISO C++11 requires whitespace after the macro name
33 | #define A/B/PROTO/EXAMPLE_H

Why are the header guards generated with "/" in them instead of "_"?

Setup Script Include Not Overriding System Protoc

I am developing on Ubuntu 20 x64 and am following the installation instructions. My system already has another version of protoc installed that is used for another dependency so I want to use the --include option in the setup.py script to specify the correct version of protoc (v21.5) but the setup script is not using the new version. I am installing protoc by unzipping this download of protoc-21.5-linux-x86_64.zip. When I unzip this file, it produces an include and bin folder. I then provide the path in the setup.py script with the include option to the include folder but the script still sees the version that was already installed on my system before. I am not sure if this is because I am installing protoc incorrectly or if the script is not able to see the executable for protoc v21.5 because the include folder doesn't contain the executable for protoc, the bin folder contains it.

Installation not working on Ubuntu - Is my protobuf installed wrong?

I'm getting the following error when I try to set things up on Ubuntu:

> python setup.py

================================================================================
|                                                                              |
| Three simple things you can do to help improve Embedded Proto:               |
|  * Give private feedback:                                                    |
|        https://EmbeddedProto.com/feedback                                    |
|  * Report an issue in public on Github:                                      |
|        https://github.com/Embedded-AMS/EmbeddedProto/issues                  |
|  * Stay up to date on Embedded Proto via our User mailing list:              |
|        https://EmbeddedProto.com/signup                                      |
|                                                                              |
================================================================================

Checking the version of Python
Checking your Protoc version.
Creating a virtual environment for Embedded Proto.
Installing requirement Python packages in the virtual environment.
Build the protobuf extension file used to include Embedded Proto custom options.
Error: Command '['protoc', '-I', 'generator', '--python_out=generator', 'embedded_proto_options.proto']' returned non-zero exit status 1.
Setup completed with success!

I installed protobuf following the instructions here: https://github.com/protocolbuffers/protobuf/blob/main/src/README.md
As a result protoc is installed /usr/local/bin and I can see all the include files in /usr/include/google/protobuf

I also tried python setup.py --include /usr/include/google/protobuf with a similar error:
Error: Command '['protoc', '-I', 'generator', '--python_out=generator', 'embedded_proto_options.proto', '-I', '/usr/include/google/protobuf']' returned non-zero exit status 1.

The install instructions for EmbeddedProto seem to assume that protobuf will be installed in the home directory and there will be a folder called "include". But I'm not sure how to install protobuf in this way... What am I doing wrong?

Repeated Message Custom Options Not Working in Python Environment

I am attempting to use the custom options for repeated messages (source) in python so that I can send data to an Arduino, but I am not able to figure out how to get the custom options to be a visible module in my python 3.8.10 installation. I was not able to find documentation in the website on how to make EmbeddedProto custom options available in my python environment.
Here is some investigation:
I have run the protoc command making sure to include the -I./generator option and the resulting python code compiles without issue and all features except the custom options work. The generated python code correctly finds the embedded_proto_options.proto file. However when I run the python code, it is not able to find embedded_proto_options_pb2 as it expects it to be a module.

test/test_example.py:16: in <module>
    from motor_driver.generated import example_pb2
motor_driver/generated/example_pb2.py:14: in <module>
    import embedded_proto_options_pb2 as embedded__proto__options__pb2
E   ModuleNotFoundError: No module named 'embedded_proto_options_pb2'

To work around this, I made a module named 'embedded_proto_options_pb2' and copied the ./generator/embedded_proto_options_pb2.py file into that module. Now when I run my code, the module is found and 'embedded__proto__options__pb2' is accessible. However I get the following error:

test/test_example.py:16: in <module>
    from motor_driver.generated import example_pb2
motor_driver/generated/example_pb2.py:17: in <module>
    DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\rexample.proto\x1a\x1c\x65mbedded_proto_options.proto\"\[rest omitted for purpose of this git issue]')
E   TypeError: Couldn't build proto file into descriptor pool: Depends on file 'embedded_proto_options.proto', but it has not been loaded

I am not sure how to load the proto file. Any suggestions to make using EmbeddedProto with python more seamless would be great.

Error executing `python setup.py` on MacOS

I'm facing the following error...

My environment:

• Python 3.11.4
• pip 23.1.2
• protoc libprotoc 23.3 installed with brew brew install protobuf

================================================================================
|                                                                              |
| Three simple things you can do to help improve Embedded Proto:               |
|  * Give private feedback:                                                    |
|        https://EmbeddedProto.com/feedback                                    |
|  * Report an issue in public on Github:                                      |
|        https://github.com/Embedded-AMS/EmbeddedProto/issues                  |
|  * Stay up to date on Embedded Proto via our User mailing list:              |
|        https://EmbeddedProto.com/signup                                      |
|                                                                              |
================================================================================

Checking the version of Python [Success]
Checking your Protoc version [Fail]
Error: 'NoneType' object has no attribute 'group'

I have fixed downgrading of the Protoc version using brew install protobuf@21 and now I have proc libprotoc 3.21.12, but this time I had to add this protoc library to my path, I don't know why, but brew couldn't handle it this time.

Expose mutable string and byte buffer and the set_length API

Currently, to assign the elements in a string or a byte buffer field, one must either use operator[] or operator=.

However, it is common in embedded world to assign the raw buffer with memcpy, str(n)cpy, s(n)printf, etc. We have to allocate another temporary buffer to contain the data, then copy to the message buffer. This causes memory usage bloating by a factor of two.

By exposing the mutable (no const qualifier) access to the internal buffer, the intermediate buffer is no longer required, hence saving memory.

In the current API, the mutable pointer could be accessed with &message[0]. However, after copying operation, the field size cannot be updated because the set_length api is private.

I suggest to expose the set_length API for the sake of the application using the string/byte internal buffer directly.

undefined reference to operator delete(void*, unsigned int)

I'm building an image with gcc-arm-none-eabi using the most recent develop branch with some of the fixed issues. It now compiles, but it doesn't link properly, because there are a lot of calls to operator delete. Is there some memory allocation happening?

/toolchains/gcc-arm-none-eabi-10-2020-q2-preview/arm-none-eabi/bin/ld: _build.DEV/_.cpp.o: in function `a::G::~G()':
_.h:56: undefined reference to `operator delete(void*, unsigned int)'
/toolchains/gcc-arm-none-eabi-10-2020-q2-preview/arm-none-eabi/bin/ld: _build.DEV/_.cpp.o: in function `a::C<17ul>::~C()':
_.h:214: undefined reference to `operator delete(void*, unsigned int)'
/toolchains/gcc-arm-none-eabi-10-2020-q2-preview/arm-none-eabi/bin/ld: _build.DEV/_.cpp.o: in function `EmbeddedProto::RepeatedFieldFixedSize<a::G, 17ul>::~RepeatedFieldFixedSize()':
/EmbeddedProto/src/RepeatedFieldFixedSize.h:64: undefined reference to `operator delete(void*, unsigned int)'
/toolchains/gcc-arm-none-eabi-10-2020-q2-preview/arm-none-eabi/bin/ld: _build.DEV/_.cpp.o: in function `EmbeddedProto::MessageSizeCalculator::~MessageSizeCalculator()':
/EmbeddedProto/src/MessageSizeCalculator.h:54: undefined reference to `operator delete(void*, unsigned int)'
/toolchains/gcc-arm-none-eabi-10-2020-q2-preview/arm-none-eabi/bin/ld: _build.DEV/Fields.cpp.o: in function `EmbeddedProto::int32::~int32()':
/EmbeddedProto/src/Fields.h:123: undefined reference to `operator delete(void*, unsigned int)'
/toolchains/gcc-arm-none-eabi-10-2020-q2-preview/arm-none-eabi/bin/ld: _build.DEV/ReadBufferSection.cpp.o:/EmbeddedProto/src/ReadBufferSection.h:63: more undefined references to `operator delete(void*, unsigned int)' follow
collect2: error: ld returned 1 exit status

Dynamically allocate memory for message string and bytes fields

Currently, this library adopts the memory model of statically sized std::array for size-variable fields like string and bytes. This benefits most embedded development workflow since no heap allocation occurs, hence the message creation is a break-or-make.

However, I am currently designing some systems involving extreme message sizes. For example, a bytes field can be as short as 1 byte of payload or as long as 100k bytes. Under current memory model, I have to create the message with the max possible size: 100k. Considering that I may also have a queue for the messages, it can easily bloat the memory usage to MBs, even if I only have a few packet with the shortest bytes field.

My suggestion is to provide a protobuf option or an optional template argument, to allow user to specify the desired internal memory container. For example, std::vector.

Introducing the dynamically allocated fields comes with the cost of potentially failed memory allocation. During encode and decode the error can be captured using the return value. However, during the message creation (assigning, modifying the message content), the possible memory allocation may fail. To capture that, the standard way is to use try-catch exception, which seems also should be avoided for embedded environment due to its overhead.

This feature may help extending the application on the platform with moderate RAM space while involving more complex communication and data exchange. For example, my ESP32-S3-WROOM-1-N8R8 has hundreds KB of internal RAM plus 8 MB of external RAM. The operating system infrastructure prevents me to use the Google implementation of protobuf. Therefore, I am doing some research to see if this library will suite my application.

What's your opinion regarding the dynamically-sized string and bytes fields?

[(EmbeddedProto.options).maxLength = xxx] causes weird memory scaling issue?

I ran into a weird issue and I'm not sure how to check whether it's an issue with my SoC (hardware) I'm using or an issue with EmbeddedProto.

To replicate the following could be tried:

Create a new message with:

message Test { repeated bytes testbytes = 1 [(EmbeddedProto.options).maxLength = 5]; }

Then in C++ do a sizeof(Options::Test); to check the size of the resulting EmbeddedProto Test class. You'll get a reasonable size.

Next try:

message Test { repeated bytes testbytes = 1 [(EmbeddedProto.options).maxLength = 512]; }

In my case the sizeof() method returned a size of 266368 bytes, which is way more than I would expect and I'm not sure what's causing it. It also caused my code to crash at a seemingly completely random point (way before the EmbeddedProto code would get called) which smells of memory leak or just the flash/RAM being too full (in my case I only have 64kb RAM).

Is maxLength somehow (silently) limited to an 8 bit unsigned int and overflowing or is something else going on?

Cannot have a 'repeated' enum field

The RepeatedField template class requires its DATA_TYPE field to be a type derived from the Field class. This rules out having a repeated enum field in a message.

e.g

enum QoS {
  none = 0;
  good = 1;
  better = 2;
}

message Subscribe {
  repeated string topics = 1
  repeated QoS qoses = 2;
}

Results in:

../../EmbeddedProto/RepeatedField.h:52:71: error: static assertion failed: A Field can only be used as template paramter.

Improve Protoc version handling

Issue Description

The current implementation assumes that Protobuf version always defines a major version, but there are cases where the major version may not always be present. As a result, the existing regular expression captures the minor number as the major and the patch number as the minor.

Existing Modifications

There have been prior attempts to handle this issue, but they do not seem to provide a reliable solution.

Context

Embedded Proto Version: 3.3.2

Expected Behavior

When encountering Protobuf versions without a major version, the version handling should correctly identify and capture major, minor, and patch numbers.

Observed Behavior

The script setup.py doesn't correctly identify and capture major, minor, and patch numbers.

Steps to Reproduce

1. Encounter a Protobuf version without a major version (e.g., "libprotoc 25.0").
2. Try to use the setup.py script.

String length template not working for nested messages

When a message (Message A) is created which contains a string, then the generated class has a template argument for the length of the string (this works fine).

If that message (Message A) is used in another message (Message B) as a repeated type then the new message has template arguments for how many times the original message (Message A) is repeated, and how long the string is in the original message (Message A). (This works fine).

However if the original message (A) is used in another message (message C) either just straight up (not repeated), or as part of a 'oneof' then the new message (C) WILL NOT have template arguments for the length of the string variable in the original message (A).
Attached is a sample proto file and the generated code using v2.2.
test_proto.txt
test_h.txt

oneof containing a message causes `--eams_out: Embedded Proto ERROR`

When compiling the following .proto

syntax = "proto3";
import "embedded_proto_options.proto";

message Periodic {
}

message BuzzerCommand {
  message Request {
    oneof buzzer_control {
      Periodic blink = 1;
    }
  }
}

with the command

protoc --plugin=protoc-gen-eams.bat --eams_out=. local_command.proto

The following error is thrown

--eams_out: Embedded Proto ERROR - Messages with repeated, string or byte fields use template parameters to define their length.For some reason it was not to add all required template parameters.

If oneof contains a message this error is thrown. When compiling this proto for other languages no error occured.

overlong varint

DeserializeVarint() does not try to detect overlong varints. It just stops processing after it has consumed N_BYTES_IN_VARINT + 1 input bytes and does not consume further input bytes even when the last byte's MSB is set.

Is this the intended behavior? I do not know enough details about the protobuf specs for this situation. Maybe even the specs do not say, but I believe the official protoc fails parsing when an overlong varint is encountered. In the following code, the output is "0 11619996266903654561 3", indicating that deserialize() succeded, but protoc --decode on the same input stream fails with "Failed to parse input."

Dummy.proto
===========
syntax = "proto3";

message Dummy {
    uint64 x = 1;
    uint32 y = 2;
}

test.cpp
========
#include <cstdint>
#include <iostream>
#include "Dummy.h"

using namespace std;

char encoding[] = {
    '\x8', '\xa1', '\xa1', '\xa1', '\xa1', '\xa1', '\xa1', '\xa1', '\xa1', '\xa1', '\xa1', '\x10',
    '\x10', '\x3'
};

class ReadBufferInterface final : public EmbeddedProto::ReadBufferInterface {
public:
    ReadBufferInterface(uint8_t const* begin, uint8_t const* end)
        : mBegin(begin), mEnd(end), mCur(begin) { }
    uint32_t get_size() const override { return mEnd - mCur; }
    uint32_t get_max_size() const override { return mEnd - mBegin; }
    bool peek(uint8_t& byte) const override {
        if (mCur == mEnd)
            return false;
        byte = *mCur;
        return true;
    }
    void advance() override {
        if (mCur != mEnd)
            ++mCur;
    }
    void advance(uint32_t const n) override {
        mCur += n;
        if (mCur > mEnd)
            mCur = mEnd;
    }
    bool pop(uint8_t& byte) override {
        if (mCur == mEnd)
            return false;
        byte = *mCur;
        ++mCur;
        return true;
    }

private:
    uint8_t const* mBegin;
    uint8_t const* mEnd;
    uint8_t const* mCur;
}; // ReadBufferInterface

int main() {
    ReadBufferInterface rbi(reinterpret_cast<uint8_t const*>(encoding),
                            reinterpret_cast<uint8_t const*>(encoding) + sizeof(encoding));
    Dummy dummy;
    auto error = dummy.deserialize(rbi);
    cout << static_cast<uint16_t>(error) << ' ' << dummy.x() << ' ' << dummy.y() << '\n';
    return 0;
};

EmbeddedProto protoc plugin only generates .h file.

Hi there,

I just downloaded EmbeddedProto to see if it will work for a project of mine, I followed the installation instructions and ran setup.py and said it ran correctly. I'm still a little unsure about what the setup.py even did though, nothing seemed to show up/change.
Now I am trying to run the command: protoc --plugin=protoc-gen-eams=Path\to\embeddedProto/protoc-gen-eams.bat -I ./ --eams_out=./ FileName.proto
But the only thing that pops up in the directory is the FileName.h file. No FileName.cpp
If you have any idea why that might be happening, or if I am misusing the command let me know. I just couldn't find that much information about what might be causing the problem so I'm asking here.

Thank you.

FieldString operator= triggers stringop-truncation error

EmbeddedProto/src/FieldStringBytes.h

Line 300 in 3fe7922

strncpy(this->get(), rhs, this->get_length());

As strlen does not count the tailing \0, the compiler check will complain that this strncpy may be truncated.

Changing this line to strncpy(this->get(), rhs, this->get_length() + 1); will suppress this warning/error.

ReadBufferInterface interface

Both overloads of advance() return void and so have no way to report an error condition, in particular, end-of-input.

Since in the recent commit a0819ae advance() started to actually be used, this commit is when the problem starts to be real.

The practical problem all this results in right now is that when skipping an unknown length-delimited field, you could skip pass the end-of-input without realizing it.

There is also the question of what to do when the number of remaining bytes is less than the requested number of bytes, whether to not advance at all or advance to the end anyway.

repeated string is interpreted as non-repeated string

Thank you for publishing an easy-to-use library.
I had a problem when trying to use repeated string.

Tested on

• EmbeddedProto: develop fb53b82
• OS: Windows10 Home / wsl

.proto

message repeated_fields 
{
  uint32 x            = 1;
  repeated uint32 y   = 2;
  uint32 z            = 3;

  string str   = 4;
  repeated string rep_str  = 5;
}

genereated code

    EmbeddedProto::uint32 x_;
    ::EmbeddedProto::RepeatedFieldFixedSize<EmbeddedProto::uint32, y_LENGTH> y_;
    EmbeddedProto::uint32 z_;
    ::EmbeddedProto::FieldString<str_LENGTH> str_;
    ::EmbeddedProto::FieldString<rep_str_LENGTH> rep_str_;

expected result

    ::EmbeddedProto::RepeatedFieldFixedSize<::EmbeddedProto::FieldString<rep_str_xxx_LENGTH>,rep_str_LENGTH> rep_str_;

Setting message field when message has oneof fails

syntax = "proto3";

message B {
  int32 e = 1;
  oneof one {
    int32 a = 2;
    int32 b = 3;
  }
}

message A { B b = 1; }

This sequence fails:

B b;
b.set_e(10);
A a;
a.set_b(b)

Output:

.h: In member function ‘void A::set_b(const B&)’:
.h:305:46: error: use of deleted function ‘B& B::operator=(const B&)’
  305 |     inline void set_b(const B& value) { b_ = value; }
      |                                              ^~~~~
.h:45:7: note: ‘B& B::operator=(const B&)’ is implicitly deleted because the default definition would be ill-formed:
   45 | class B final: public ::EmbeddedProto::MessageInterface
      |       ^
.h:45:7: error: use of deleted function ‘B::one& B::one::operator=(const B::one&)’
.h:240:11: note: ‘B::one& B::one::operator=(const B::one&)’ is implicitly deleted because the default definition would be ill-formed:
  240 |     union one
      |           ^~~
.h:244:28: error: union member ‘B::one::a_’ with non-trivial ‘constexpr EmbeddedProto::int32& EmbeddedProto::int32::operator=(const EmbeddedProto::int32&)’
  244 |       EmbeddedProto::int32 a_;
      |                            ^~
.h:245:28: error: union member ‘B::one::b_’ with non-trivial ‘constexpr EmbeddedProto::int32& EmbeddedProto::int32::operator=(const EmbeddedProto::int32&)’
  245 |       EmbeddedProto::int32 b_;
      |                            ^~
.h: In member function ‘void A::set_b(const B&&)’:
.h:306:47: error: use of deleted function ‘B& B::operator=(const B&)’
  306 |     inline void set_b(const B&& value) { b_ = value; }

RepeatedFieldFixedSize.h compares signed and unsigned integers in get_const()

The index argument to get_const() within RepeatedFieldFixedSize.h is a signed integer and it is being compared to an unsigned integer. This causes compiler warnings. Updating the 'index' argument to be 'const uint32_t' resolves this. Additionally, the index argument is an index into an array; if a negative value is passed in, then the range check against the current length will still pass and result in data corruption.

Order of proto messages in .proto matters

The order of proto message definition in the .proto file matters. If a proto message is defined after it is used, the generated code fails to compile. I have seen 2 concrete issues so far:

1. No forward declarations happen
2. template arguments for fields are omitted

1. No forward declarations happen

message A {
  B b = 1;
}

message B {
  int32 val = 1;
}

This fails because B hasn't been declared yet, and class A references an undefined label B.

2. template arguments for fields are omitted

message D {
  C c = 1;
}

message C {
  repeated int32 list = 1;
}

This fails because of #1, but the template arguments for C are not generated. Instead, this gets generated:

class D final: public ::EmbeddedProto::MessageInterface
{
  public:
    D() :
        c_()
    {

    };
    ~D() override = default;

    enum class id
    {
      NOT_SET = 0,
      C = 1
    };

    inline const C& c() const { return c_; }
    inline void clear_c() { c_.clear(); }
    inline void set_c(const C& value) { c_ = value; }
    inline void set_c(const C&& value) { c_ = value; }
    inline C& mutable_c() { return c_; }
    inline const C& get_c() const { return c_; }

    ::EmbeddedProto::Error serialize(::EmbeddedProto::WriteBufferInterface& buffer) const final
    {
      ::EmbeddedProto::Error return_value = ::EmbeddedProto::Error::NO_ERRORS;

      if(::EmbeddedProto::Error::NO_ERRORS == return_value)
      {
        return_value = c_.serialize_with_id(static_cast<uint32_t>(id::C), buffer);
      }
       

      return return_value;
    };

    ::EmbeddedProto::Error deserialize(::EmbeddedProto::ReadBufferInterface& buffer) final
    {
      ::EmbeddedProto::Error return_value = ::EmbeddedProto::Error::NO_ERRORS;
      ::EmbeddedProto::WireFormatter::WireType wire_type;
      uint32_t id_number = 0;

      ::EmbeddedProto::Error tag_value = ::EmbeddedProto::WireFormatter::DeserializeTag(buffer, wire_type, id_number);
      while((::EmbeddedProto::Error::NO_ERRORS == return_value) && (::EmbeddedProto::Error::NO_ERRORS == tag_value))
      {
        switch(id_number)
        {
          case static_cast<uint32_t>(id::C):
          {
            if(::EmbeddedProto::WireFormatter::WireType::LENGTH_DELIMITED == wire_type)
            {
              uint32_t size;
              return_value = ::EmbeddedProto::WireFormatter::DeserializeVarint(buffer, size);
              ::EmbeddedProto::ReadBufferSection bufferSection(buffer, size);
              if(::EmbeddedProto::Error::NO_ERRORS == return_value)
              {
                return_value = mutable_c().deserialize(bufferSection);
              }
            }
            else
            {
              // Wire type does not match field.
              return_value = ::EmbeddedProto::Error::INVALID_WIRETYPE;
            } 
            break;
          }

          default:
            break;
        }
        
        if(::EmbeddedProto::Error::NO_ERRORS == return_value)
        {
            // Read the next tag.
            tag_value = ::EmbeddedProto::WireFormatter::DeserializeTag(buffer, wire_type, id_number);
        }
      }

      // When an error was detect while reading the tag but no other errors where found, set it in the return value.
      if((::EmbeddedProto::Error::NO_ERRORS == return_value)
         && (::EmbeddedProto::Error::NO_ERRORS != tag_value)
         && (::EmbeddedProto::Error::END_OF_BUFFER != tag_value)) // The end of the buffer is not an array in this case.
      {
        return_value = tag_value;
      }

      return return_value;
    };

    void clear() final
    {
      clear_c();
    }

  private:

    C c_;

};

Note that C should be C<c_LIST_LENGTH> and the template should be passed through class D.

"end of namespace" comment out of order

This doesn't really matter, but I figured I'd point it out anyway

Namespaces look like this:

namespace a {
namespace b {
} // End of namespace a
} // End of namespace b

Note the comments at the end are flipped

Optimize `oneof`

Is it really necessary to clear to contents of the field when clearing the oneof:

EmbeddedProto/generator/templates/TypeOneof.h

Line 97 in 7ea7b8b

{{field.get_variable_name()}}.set(0);

or might it be enough to leave whatever data is in there as it will be overwritten by the next set() operation before this field can be used again anyways (maybe we have to move the clear() call in the set() call when it is not guaranteed that everything will be overwritten by the next set() call). Especially for large byte fields (e.g. in the range kilobytes) this might have a big performance impact on embedded devices!

Trouble getting venv to work on Ubuntu 18.04

Setup steps:
Installed python3, pip, venv, protobuf, git, cmake, etc.
Ran setup.sh script

Problem:
All paths in protoc-gen-eams are relative. When generating protos from a different directory, these paths all fail to resolve.

Solution:
Add $(dirname $0) in front of both ./venv/bin/python and generator/protoc-gen-eams.py