kube-rs / k8s-pb Goto Github PK
View Code? Open in Web Editor NEWPlaying with Kubernetes protobufs
License: Apache License 2.0
Playing with Kubernetes protobufs
License: Apache License 2.0
I'd like to write cache apiserver/proxy to offload "list-all-pods" requests from non-critical clients. So k8s_openapi json is not suitable for my cases.
Current golang PoC is suffer from go runtime cost and protobuf/json marshaling I have strong interests try/evaluate rust solution.
Any example would be good and appreciative, thanks!
Was looking into ways people do codegen in rust and was trying to identify some pros/cons between the various approaches. I suspect this is a simple question, but just wanted to make sure.
Seen 3 different categories:
quote
It's possible at a large scale the templating system will work, but it feels pretty hard to maintain. Big input changes and suddenly you have some awful compile errors to debug. Looked at minjinja for its light dependency tree. This will just silently unwrap to empty strings on unknown mistyped keys in context objects (only erroring if trying to traverse through missing objects), so awful for maintainability. Maybe a bigger engine can work. But this approach sounds very brittle to me.
For typed api builders, there is carllerche/codegen, which is an interesting idea, but it's abandoned. Prs with features ignored for a year - most activity people self-closing. Not sure if there are even better options for this.
By far the most scalable setup to me seems to be using quote
with a module/fn per "thing we want to generate". Will give us compile errors on misuse and we are already pretty familiar with it from kube-derive.
We need to handle the envelope wrapper application/vnd.kubernetes.protobuf
(4 bytes prefix + Unknown
message). This makes the message self describing.
It should be pretty similar to how we're handling JSON.
https://github.com/kubernetes/api/blob/master/README.md#recommended-use
If you want to store or interact with proto-formatted Kubernetes API objects, we recommend using the "official" serialization stack in
k8s.io/apimachinery
. Directly serializing these types to proto will not result in data that matches the wire format or is compatible with other kubernetes ecosystem tools. The reason is that the wire format includes a magic prefix and an envelope proto. Please see: kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encodingFor the same reason, we do not recommend embedding these proto objects within your own proto definitions. It is better to store Kubernetes objects as byte arrays, in the wire format, which is self-describing. This permits you to use either JSON or binary (proto) wire formats without code changes. It will be difficult for you to operate on both Custom Resources and built-in types otherwise.
https://kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding
Kubernetes uses an envelope wrapper to encode Protobuf responses. That wrapper starts with a 4 byte magic number to help identify content in disk or in etcd as Protobuf (as opposed to JSON), and then is followed by a Protobuf encoded wrapper message, which describes the encoding and type of the underlying object and then contains the object.
A four byte magic number prefix:
Bytes 0-3: "k8s\x00" [0x6b, 0x38, 0x73, 0x00]
An encoded Protobuf message with the following IDL:
message Unknown {
// typeMeta should have the string values for "kind" and "apiVersion" as set on the JSON object
optional TypeMeta typeMeta = 1;
// raw will hold the complete serialized object in protobuf. See the protobuf definitions in the client libraries for a given kind.
optional bytes raw = 2;
// contentEncoding is encoding used for the raw data. Unspecified means no encoding.
optional string contentEncoding = 3;
// contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf and is usually
// omitted.
optional string contentType = 4;
}
message TypeMeta {
// apiVersion is the group/version for this type
optional string apiVersion = 1;
// kind is the name of the object schema. A protobuf definition should exist for this object.
optional string kind = 2;
}
Generated Rust:
k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.runtime.rs
Lines 87 to 104 in 7b4bf26
k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.runtime.rs
Lines 68 to 76 in 7b4bf26
This is a possible feature for a much later if we get things working.
One of the things that I found was ugly with k8s-openapi
was the need to structs everywhere with optionals so you ended up with ..Default::default()
everywhere (leading people to use json!
)
Maybe it's worth exploring builders everywhere: https://github.com/idanarye/rust-typed-builder seems to handle options well and require you fill in only required fields (which in kubernetes should be very few). Teo might also have opinions here, he wrote a builder crate.
What we could do in the future (if this gets off the ground) is to optionally #[derive(TypeBuilder)]
for the structs.
Possibly a bit heavy, I know Arnav rejected proposals at more ergonomic apis before because of crate size (already heavy enough to compile everything), but that's possibly something we could aid by providing ways to limit the size of the crate (like only exposing the structs for the api groups that you are interested in for instance).
Anyway. This is a note to self for later.
Just an idea for Arnavion/k8s-openapi#77. Finding the dependencies between modules should be possible because each package has imports (FileDescriptorProto
has dependency
field).
k8s-pb/k8s-pb-codegen/protos/api/core/v1/generated.proto
Lines 22 to 28 in 7b4bf26
We also don't need to worry about multiplying that by version features if we version this using Kubernetes version (#10).
We should not export version features in this crate ala k8s-openapi
:
They really annoying to use with resolver2 workspaces -> people have to select a version, even though they are not using anything other than metadata for some kube_core
structs.
They send a false message that hardcoding the kubernetes version in your toml is necessary, when in reality, kubernetes is not breaking any apis (they release new versions of them and remove deprecated ones).
It also ends up being impossible to run workspace tests on a matrix of kubernetes version in kube-rs
when the versions are features because dev-deps need to select one version, and then we can't unselect the dev-dep version for tests.
We COULD re-export the features ourselves (and re-export the structs), but they end up everywhere in the tomls:
# Supported kubernetes versions
v1_17 = ["k8s-openapi/v1_17", "kube-core/v1_17", "kube-client/v1_17", "kube-runtime/v1_17"]
v1_18 = ["k8s-openapi/v1_18", "kube-core/v1_18", "kube-client/v1_18", "kube-runtime/v1_18"]
v1_19 = ["k8s-openapi/v1_19", "kube-core/v1_19", "kube-client/v1_19", "kube-runtime/v1_19"]
v1_20 = ["k8s-openapi/v1_20", "kube-core/v1_20", "kube-client/v1_20", "kube-runtime/v1_20"]
v1_21 = ["k8s-openapi/v1_21", "kube-core/v1_21", "kube-client/v1_21", "kube-runtime/v1_21"]
v1_22 = ["k8s-openapi/v1_22", "kube-core/v1_22", "kube-client/v1_22", "kube-runtime/v1_22"]
and we still have to set a default version for us to run cargo build
with this way (otherwise it would be super impractical for us). If we set a default version feature, then we need to unselect them for matrix CI, and cascade non-defaults everywhere, leading to a nightmare of features and dev-deps again.
We should define our own client-go
like compatibility matrix.
This heavily matches what we want. It reflects the guarantees we really have (apis either disappear because they are deprecated+removed, or are not in that version of kube because we didn't use a new enough codegen).
That way we simply announce that kube has at least these api features available, and newer versions of kube might lack some deprecated apis that were removed by kubernetes in this version.
A thing that can be done in parallel to the implementation of the code-generation, is to sketch out the traits we need to implement in the codegen, and whether we need to rethink them a bit.
There is currently a bit of a disparity between k8s-openapi
traits and kube_core
's traits, and it would be nice if the traits we used in kube
, were the traits defined either by kube_core
or something here.
So here's a rough list of generics I think we should aim to implement / refine for this project to be successful:
Need at least these two to be able to implement kube::Resource
upstream:
k8s-openapi::ResourceScope
equivalent. from old issue kube-rs/kube#194 - done in #25k8s-openapi::Resource
equivalent - #14 + #25HasSpec
and HasStatus
: https://github.com/kube-rs/kube-rs/blob/master/kube-core/src/object.rs - #14HasConditions
: needed for things like kube-rs/kube#679 - #14HasMetadata
:: an analogue of the k8s_openapi::Metadata
trait - #14Here? Or separate repo? I'm leaning towards defining a kube-traits
crate within the kube-rs
org and implementing it here.
But open to ideas. Also, anything missing from this list?
These could maybe be done later for even more generic action.
Log
- these should probably not live in kube-client
)k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.apis.meta.v1.rs
Lines 1031 to 1051 in 7b4bf26
codegenrs
makes it easy to get rid of code-gen inbuild.rs
, reducing build times for your crate and those that depend on itThis is done by:
- Creating a child
[[bin]]
crate that does code-gen usingcodegenrs
- Do one-time code-gen and commit it
- Run the
--check
step in your CI to ensure your code-gen is neither out of
date or been human edited.
Seems to support using rustfmt
to format the generated code. https://github.com/crate-ci/imperative/blob/f7955ed1deda8abc8aeacc3a41aca23b3acc47d0/codegen/src/main.rs#L99
Currently subresources, while they are generated (see i.e. Eviction), do not get a Resource
impl because they do not get transformed via the jq transform.
It exists in the swagger.json as:
"io.k8s.api.policy.v1.Eviction": {
"description": "Eviction evicts a pod from its node subject to certain policies and safety constraints. This is a subresource of Pod. A request to cause such an eviction is created by POSTing to .../pods/<pod name>/evictions.",
"properties": {
"apiVersion": {
"description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
"type": "string"
},
"deleteOptions": {
"$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions",
"description": "DeleteOptions may be provided"
},
"kind": {
"description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
"type": "string"
},
"metadata": {
"$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta",
"description": "ObjectMeta describes the pod that is being evicted."
}
},
"type": "object",
"x-kubernetes-group-version-kind": [
{
"group": "policy",
"kind": "Eviction",
"version": "v1"
}
]
},
We need to capture this somehow into the transformed json.
Some irregular camel case to snake case conversions are failing (e.g., clusterIPs
becomes cluster_i_ps
). But prost_build
doesn't provide a way to fix this at the moment.
Initially as a hidden PoC to assess compatibility and see what unknown-unknowns we need to handle.
Expected work (all under a kube/pb
feature):
k8s-pb
as an optional dependency on a branch (pin a git dependency while working)kube::Resource
for k8s-pb::Resource
implementors in kube_core
- can use HasMetadata
as well (via #4)If there are lacks on the k8s-pb
side we can address it here as we go along.
Work we also should do (but could be avoided for a first PoC):
k8s-openapi
by using k8s-pb
structs instead when in feature (lots of fiddly work)Currently there's a single entry pkg
in most of the exported main modules:
but it's not present in the main https://github.com/kube-rs/k8s-pb/tree/main/k8s-pb/src/api .
We should slice away this needless export path to make usage a little more ergonomic.
Prost also converts aggressive PascalACRONYMCase names (with allcaps somewhere inside) like APIService
to ApiService
.
In general, this is a good improvement to the rust conventions, but, like #2, it has some problems:
ApiResource
is an abstraction for APIResource
in discovery/coreApiGroup
similarly similarly for APIGroup
in discoveryit's already a bit awkward that we convert their camelCased variable names to snake_case, this will also break some of the main structs.
personally, i don't really think 2. is a problem (everyone has been happy with the snake_case conversion in in k8s-openapi despite my earlier reservations), and this extra change only affects a small amount of structs who uses a decidedly bad naming convention.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.