ed-xcf / protobuf2pydantic Goto Github PK
View Code? Open in Web Editor NEWgenerate pydantic model by protobuf.pb2 file
License: MIT License
generate pydantic model by protobuf.pb2 file
License: MIT License
Hi there,
Very nice pkg, I was testing it and quickly (for my use case [1]) I needed to convert from proto
instance to pydantic
, then I was wondering what do you think about adding a helper method to the generated classes such .from_proto
? if you agree in the idea I can jump in in the implementation (or if you prefer implementing all good) .
[1] just playing in a proof of concept project that from a proto
definition I generate the following services rpc
(grpc), rest
(fastapi), graphql, I basically 1) compile the proto
and add the grpc server 2) use your lib to generate pydantic that empowers fastapi 3) from the generated pydantic
I use pydantic2graphene
to expose for graphql
from protobuf2pydantic import msg2py
from pydantic import validator
import transaction_pb2
class AmountResponse(msg2py(transaction_pb2.AmountResponse)):
@validator("amount")
def non_negative(cls, v):
assert v >= 0
return v
msg_proto = transaction_pb2.AmountResponse(amount=10.01, currency="USD")
amount_pydantic = AmountResponse.from_proto(msg_proto)
Hello,
Thanks a lot for this library, it is perfect to interface ProtoBufs and FastApi.
I got troubles with the following proto:
syntax = "proto3";
message Entity {
int32 start=1;
int32 stop=2;
string type=3;
string text=4;
string value=5;
}
message EntityList {
repeated Entity entities = 1;
}
message EntityOutput {
map<string, EntityList> entities_per_lang = 1;
}
Got a KeyError : 11 as the field in the dict is another proto field.
This could be solved by taking the corresponding class.
(I could make a PR but as the code is small I'm pasting it here. If it's easier for you I can make a PR)
Changing the convert field function to
def convert_field(level: int, field: FieldDescriptor) -> str:
level += 1
field_type = field.type
field_label = field.label
was_mapping = False
extra = None
if field_type == FieldDescriptor.TYPE_ENUM:
enum_type: EnumDescriptor = field.enum_type
type_statement = enum_type.name
class_statement = f"{tab * level}class {enum_type.name}(IntEnum):"
field_statements = map(
lambda value: f"{tab * (level + 1)}{value.name} = {value.index}",
enum_type.values,
)
extra = linesep.join([class_statement, *field_statements])
factory = "int"
elif field_type == FieldDescriptor.TYPE_MESSAGE:
type_statement: str = field.message_type.name
if type_statement.endswith("Entry"):
key, value = field.message_type.fields # type: FieldDescriptor
if value.type != 11:
type_statement = f"Dict[{m(key)}, {m(value)}]"
else:
was_mapping = True
type_statement = f"Dict[{m(key)}, {value.message_type.name}]"
factory = "dict"
elif type_statement == "Struct":
type_statement = "Dict[str, Any]"
factory = "dict"
else:
extra = msg2pydantic(level, field.message_type)
factory = type_statement
else:
type_statement = m(field)
factory = type_statement
if field_label == FieldDescriptor.LABEL_REPEATED and not was_mapping:
type_statement = f"List[{type_statement}]"
factory = "list"
default_statement = f" = Field(default_factory={factory})"
if field_label == FieldDescriptor.LABEL_REQUIRED:
default_statement = ""
field_statement = f"{tab * level}{field.name}: {type_statement}{default_statement}"
if not extra:
return field_statement
return linesep + extra + one_line + field_statement
Have a great day
Hi
I think there's some issue with the CLI.
I did the following:
python3.9 -m venv .venv
source .venv/bin/activate
pip3 install protobuf2pydantic
pb2py --help
and I got:
Traceback (most recent call last):
File "/Users/my_user/repos/tmp/.venv/bin/pb2py", line 5, in <module>
from protobuf2pydantic.main import app
File "/Users/my_user/repos/tmp/.venv/lib/python3.9/site-packages/protobuf2pydantic/__init__.py", line 11, in <module>
from protobuf2pydantic.biz import msg2pydantic
File "/Users/my_user/repos/tmp/.venv/lib/python3.9/site-packages/protobuf2pydantic/biz.py", line 64, in <module>
def convert_field(level: int, field: FieldDescriptor) -> str | None:
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
I'm getting a similar error when I run pb2py /path/to/protobuf/dummy_pb2.py > wow.py
command, and when I simply import protobuf2pydantic
.
Note the issue seem to exist in the newest version only, 2024.1.8
.
The version one before it 2023.12.21
seem to be working fine.
I'm using Python 3.9.13, on a Mac with Sonoma 14.1 OS, with protobuf2pydantic
version 2024.1.8
In the below reference, you make the value number in the generated Pydantic model equal to the index of the elements in enum. This can cause problems when enum set in the proto file is not in a running number.
It can be solved by placing value.number
instead of value.index
.
Hi.
I've just found your tool and seems magic :) very cool
as I am working with a "big" schema definition (~100 .proto files, distributed in a 3/4 level nested tree) I was wondering if you could enhance pb2py
to support the input from a folder (and implicitly convert all the *_pb2.py
files inside) and output to a target folder (and not on stdout)
at the moment I am putting in place a shell script based on find
, mkdir -p
and redirection, but it's very fragile and I was wondering if you could help on the tool side
We tried to upgrade from version 2023.12.21 to 2024.4.18 (the interim releases gave us the problems that have subsequently been fixed in the latest version).
For some reason, the output files are missing large chunks of code. Here's an example:
enum EventType {
EVENT_TYPE_UNSPECIFIED = 0;
EVENT_TYPE_ONE = 1;
EVENT_TYPE_TWO = 2;
}
message GetEventsRequest {
uint32 limit = 1;
string status_filter = 2;
EventType events_filter = 3;
}
In the previous version, the output is as follows:
class GetEventsRequest(BaseModel):
limit: int = Field(default_factory=int)
status_filter: str = Field(default_factory=str)
class EventType(IntEnum):
EVENT_TYPE_UNSPECIFIED = 0
EVENT_TYPE_ONE = 1
EVENT_TYPE_TWO = 2
events_filter: EventType = Field(default_factory=int)
In the latest version, the output is as follows:
class GetEventsRequest(BaseModel):
tenant_id: str = Field(default_factory=str)
limit: int = Field(default_factory=int)
status_filter: str = Field(default_factory=str)
So the event type class and events filter just got dropped.
In our response method, the same thing happened and all we were left with is the following, which isn't just wrong but invalid:
class GetEventsResponse(BaseModel):
My naive suspicion is that this is related to the fix with the inner class indentation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.