kube-rs / k8s-pb Goto Github PK

View Code? Open in Web Editor NEW

14.0 8.0 5.0 1.79 MB

Playing with Kubernetes protobufs

License: Apache License 2.0

Rust 99.50% jq 0.38% Just 0.12%

kubernetes protobuf rust

k8s-pb's Introduction

k8s-pb

Kubernetes protobuf bindings for kube-rs. WIP. Not yet useable as is.

Usage

This library is not currently usable from kube. For now consider these structs a reference location for Kubernetes structs that are not found in k8s-openapi. See docs.rs/k8s-pb.

Build Process

The code generation process consists of 3 steps;

just protos - download and patch protobufs
just swagger - download and transform openapi schema
just codegen - combine info and build with prost

The k8s-pb crate is generated as a result of this process and then published periodically.

just protos

Obtains the version pinned protobufs from upstream:

then does minor transforms to prepare them for building.

just swagger

Obtains the version pinned swagger openapi schema from upstream:

https://github.com/kubernetes/kubernetes/tree/master/api/openapi-spec

then applies any needed patches before transforming the schema into a shorter json file containing desired generic information.

This json file complements the protos with type type properties needed for trait implementations.

just codegen

Runs main.rs, using the outputs from the swagger and protobuf recipes above. In particular;

The protos are built with prost via protoc and provides a FileDescriptorSet via Config::file_descriptor_set_path.
The transformed swagger result json is deserialized through lib.rs into a HashMap<String, Resource> where the string is a GVK string.
We loop through all the generated modules and
- inject generics from this hashmap via append_trait_impl
- in memory rustfmt before finalising
Generate a module tree from the seen modules in the loop
Attach our implemented traits to the generated lib.rs

Build Dependencies

k8s-pb's People

Contributors

Stargazers

Watchers

Forkers

clux doytsujin ynqa bryantbiggs

k8s-pb's Issues

Support subresources

Currently subresources, while they are generated (see i.e. Eviction), do not get a Resource impl because they do not get transformed via the jq transform.

It exists in the swagger.json as:

    "io.k8s.api.policy.v1.Eviction": {
      "description": "Eviction evicts a pod from its node subject to certain policies and safety constraints. This is a subresource of Pod.  A request to cause such an eviction is created by POSTing to .../pods/<pod name>/evictions.",
      "properties": {
        "apiVersion": {
          "description": "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
          "type": "string"
        },
        "deleteOptions": {
          "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.DeleteOptions",
          "description": "DeleteOptions may be provided"
        },
        "kind": {
          "description": "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
          "type": "string"
        },
        "metadata": {
          "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta",
          "description": "ObjectMeta describes the pod that is being evicted."
        }
      },
      "type": "object",
      "x-kubernetes-group-version-kind": [
        {
          "group": "policy",
          "kind": "Eviction",
          "version": "v1"
        }
      ]
    },

We need to capture this somehow into the transformed json.

Support selecting API groups to build

Just an idea for Arnavion/k8s-openapi#77. Finding the dependencies between modules should be possible because each package has imports (FileDescriptorProto has dependency field).

k8s-pb/k8s-pb-codegen/protos/api/core/v1/generated.proto

Lines 22 to 28 in 7b4bf26

 package api.core.v1; 

 import "apimachinery/pkg/api/resource/generated.proto"; 

 import "apimachinery/pkg/apis/meta/v1/generated.proto"; 

 import "apimachinery/pkg/runtime/generated.proto"; 

 import "apimachinery/pkg/runtime/schema/generated.proto"; 

 import "apimachinery/pkg/util/intstr/generated.proto";

We also don't need to worry about multiplying that by version features if we version this using Kubernetes version (#10).

Use `codegenrs`?

codegenrs makes it easy to get rid of code-gen in build.rs, reducing build times for your crate and those that depend on it

This is done by:

Creating a child [[bin]] crate that does code-gen using codegenrs

Do one-time code-gen and commit it

Run the --check step in your CI to ensure your code-gen is neither out of
date or been human edited.

https://github.com/crate-ci/codegenrs

Seems to support using rustfmt to format the generated code. https://github.com/crate-ci/imperative/blob/f7955ed1deda8abc8aeacc3a41aca23b3acc47d0/codegen/src/main.rs#L99

Crate versioning

We should not export version features in this crate ala k8s-openapi:

They really annoying to use with resolver2 workspaces -> people have to select a version, even though they are not using anything other than metadata for some kube_core structs.
They send a false message that hardcoding the kubernetes version in your toml is necessary, when in reality, kubernetes is not breaking any apis (they release new versions of them and remove deprecated ones).
It also ends up being impossible to run workspace tests on a matrix of kubernetes version in kube-rs when the versions are features because dev-deps need to select one version, and then we can't unselect the dev-dep version for tests.

We COULD re-export the features ourselves (and re-export the structs), but they end up everywhere in the tomls:

kube cargo.toml

# Supported kubernetes versions
v1_17 = ["k8s-openapi/v1_17", "kube-core/v1_17", "kube-client/v1_17", "kube-runtime/v1_17"]
v1_18 = ["k8s-openapi/v1_18", "kube-core/v1_18", "kube-client/v1_18", "kube-runtime/v1_18"]
v1_19 = ["k8s-openapi/v1_19", "kube-core/v1_19", "kube-client/v1_19", "kube-runtime/v1_19"]
v1_20 = ["k8s-openapi/v1_20", "kube-core/v1_20", "kube-client/v1_20", "kube-runtime/v1_20"]
v1_21 = ["k8s-openapi/v1_21", "kube-core/v1_21", "kube-client/v1_21", "kube-runtime/v1_21"]
v1_22 = ["k8s-openapi/v1_22", "kube-core/v1_22", "kube-client/v1_22", "kube-runtime/v1_22"]

and we still have to set a default version for us to run cargo build with this way (otherwise it would be super impractical for us). If we set a default version feature, then we need to unselect them for matrix CI, and cascade non-defaults everywhere, leading to a nightmare of features and dev-deps again.

Proposal

We should define our own client-go like compatibility matrix.
This heavily matches what we want. It reflects the guarantees we really have (apis either disappear because they are deprecated+removed, or are not in that version of kube because we didn't use a new enough codegen).

That way we simply announce that kube has at least these api features available, and newer versions of kube might lack some deprecated apis that were removed by kubernetes in this version.

optional builder patterns

This is a possible feature for a much later if we get things working.

One of the things that I found was ugly with k8s-openapi was the need to structs everywhere with optionals so you ended up with ..Default::default() everywhere (leading people to use json!)

Maybe it's worth exploring builders everywhere: https://github.com/idanarye/rust-typed-builder seems to handle options well and require you fill in only required fields (which in kubernetes should be very few). Teo might also have opinions here, he wrote a builder crate.

What we could do in the future (if this gets off the ground) is to optionally #[derive(TypeBuilder)] for the structs.

Possibly a bit heavy, I know Arnav rejected proposals at more ergonomic apis before because of crate size (already heavy enough to compile everything), but that's possibly something we could aid by providing ways to limit the size of the crate (like only exposing the structs for the api groups that you are interested in for instance).

Anyway. This is a note to self for later.

Exclude `pkg` from exported module tree

Currently there's a single entry pkg in most of the exported main modules:

but it's not present in the main https://github.com/kube-rs/k8s-pb/tree/main/k8s-pb/src/api .

We should slice away this needless export path to make usage a little more ergonomic.

`Time`/`MicroTime` needs a wrapper

k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.apis.meta.v1.rs

Lines 1031 to 1051 in 7b4bf26

 /// Time is a wrapper around time.Time which supports correct 

 /// marshaling to YAML and JSON. Wrappers are provided for many 

 /// of the factory methods that the time package offers. 

 /// 

 /// +protobuf.options.marshal=false 

 /// +protobuf.as=Timestamp 

 /// +protobuf.options.(gogoproto.goproto_stringer)=false 

 #[derive(Clone, PartialEq, ::prost::Message)] 

 pub struct Time { 

 /// Represents seconds of UTC time since Unix epoch 

 /// 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to 

 /// 9999-12-31T23:59:59Z inclusive. 

 #[prost(int64, optional, tag="1")] 

 pub seconds: ::core::option::Option<i64>, 

 /// Non-negative fractions of a second at nanosecond resolution. Negative 

 /// second values with fractions must still have non-negative nanos values 

 /// that count forward in time. Must be from 0 to 999,999,999 

 /// inclusive. This field may be limited in precision depending on context. 

 #[prost(int32, optional, tag="2")] 

 pub nanos: ::core::option::Option<i32>, 

 }

Handle irregular camelCase to snake_case conversion

Some irregular camel case to snake case conversions are failing (e.g., clusterIPs becomes cluster_i_ps). But prost_build doesn't provide a way to fix this at the moment.

Integrate with `kube` under a feature

Initially as a hidden PoC to assess compatibility and see what unknown-unknowns we need to handle.

Expected work (all under a kube/pb feature):

add k8s-pb as an optional dependency on a branch (pin a git dependency while working)
implement kube::Resource for k8s-pb::Resource implementors in kube_core - can use HasMetadata as well (via #4)
see if we can do querying outside protobufs
try to add the envelope wrapper (via #6) so we can do actual protobuf queries in kube
add a small example here to show how to use it (#28)

If there are lacks on the k8s-pb side we can address it here as we go along.

Work we also should do (but could be avoided for a first PoC):

figure out if we can feature select out k8s-openapi by using k8s-pb structs instead when in feature (lots of fiddly work)

Define and implement equivalent `k8s-openapi` traits

A thing that can be done in parallel to the implementation of the code-generation, is to sketch out the traits we need to implement in the codegen, and whether we need to rethink them a bit.

There is currently a bit of a disparity between k8s-openapi traits and kube_core's traits, and it would be nice if the traits we used in kube, were the traits defined either by kube_core or something here.

So here's a rough list of generics I think we should aim to implement / refine for this project to be successful:

wanted traits

Need at least these two to be able to implement kube::Resource upstream:

k8s-openapi::ResourceScope equivalent. from old issue kube-rs/kube#194 - done in #25
k8s-openapi::Resource equivalent - #14 + #25

getter traits

HasSpec and HasStatus: https://github.com/kube-rs/kube-rs/blob/master/kube-core/src/object.rs - #14
HasConditions: needed for things like kube-rs/kube#679 - #14
HasMetadata :: an analogue of the k8s_openapi::Metadata trait - #14
There are probably room for more things, but we should decide where generics should live.

Here? Or separate repo? I'm leaning towards defining a kube-traits crate within the kube-rs org and implementing it here.
But open to ideas. Also, anything missing from this list?

Descoped

These could maybe be done later for even more generic action.

verb traits for subresources (e.g. Log - these should probably not live in kube-client)
verb traits for normal resources - enables neokubism kube-rs/kube#594

Figure out how to best generate the generics

Was looking into ways people do codegen in rust and was trying to identify some pros/cons between the various approaches. I suspect this is a simple question, but just wanted to make sure.

Seen 3 different categories:

direct templating using some dependency intended for html templates
some typed api builder for code
quote

It's possible at a large scale the templating system will work, but it feels pretty hard to maintain. Big input changes and suddenly you have some awful compile errors to debug. Looked at minjinja for its light dependency tree. This will just silently unwrap to empty strings on unknown mistyped keys in context objects (only erroring if trying to traverse through missing objects), so awful for maintainability. Maybe a bigger engine can work. But this approach sounds very brittle to me.

For typed api builders, there is carllerche/codegen, which is an interesting idea, but it's abandoned. Prs with features ignored for a year - most activity people self-closing. Not sure if there are even better options for this.

By far the most scalable setup to me seems to be using quote with a module/fn per "thing we want to generate". Will give us compile errors on misuse and we are already pretty familiar with it from kube-derive.

any example?

I'd like to write cache apiserver/proxy to offload "list-all-pods" requests from non-critical clients. So k8s_openapi json is not suitable for my cases.

Current golang PoC is suffer from go runtime cost and protobuf/json marshaling I have strong interests try/evaluate rust solution.

Any example would be good and appreciative, thanks!

`IntOrString` needs a wrapper

k8s-pb/k8s-pb-codegen/protos/apimachinery/pkg/util/intstr/generated.proto

Lines 36 to 42 in 7b4bf26

 message IntOrString { 

 optional int64 type = 1; 

 optional int32 intVal = 2; 

 optional string strVal = 3; 

 }

PascalCase conversion on structs in codegen

Prost also converts aggressive PascalACRONYMCase names (with allcaps somewhere inside) like APIService to ApiService.
In general, this is a good improvement to the rust conventions, but, like #2, it has some problems:

1. have to disambiguate / break names used in kube

ApiResource is an abstraction for APIResource in discovery/core
ApiGroup similarly similarly for APIGroup in discovery

2. we break even further from official names

it's already a bit awkward that we convert their camelCased variable names to snake_case, this will also break some of the main structs.

personally, i don't really think 2. is a problem (everyone has been happy with the snake_case conversion in in k8s-openapi despite my earlier reservations), and this extra change only affects a small amount of structs who uses a decidedly bad naming convention.

Handling Envelope Wrapper

We need to handle the envelope wrapper application/vnd.kubernetes.protobuf (4 bytes prefix + Unknown message). This makes the message self describing.

It should be pretty similar to how we're handling JSON.

https://github.com/kubernetes/api/blob/master/README.md#recommended-use

If you want to store or interact with proto-formatted Kubernetes API objects, we recommend using the "official" serialization stack in k8s.io/apimachinery. Directly serializing these types to proto will not result in data that matches the wire format or is compatible with other kubernetes ecosystem tools. The reason is that the wire format includes a magic prefix and an envelope proto. Please see: kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding

For the same reason, we do not recommend embedding these proto objects within your own proto definitions. It is better to store Kubernetes objects as byte arrays, in the wire format, which is self-describing. This permits you to use either JSON or binary (proto) wire formats without code changes. It will be difficult for you to operate on both Custom Resources and built-in types otherwise.

https://kubernetes.io/docs/reference/using-api/api-concepts/#protobuf-encoding

Kubernetes uses an envelope wrapper to encode Protobuf responses. That wrapper starts with a 4 byte magic number to help identify content in disk or in etcd as Protobuf (as opposed to JSON), and then is followed by a Protobuf encoded wrapper message, which describes the encoding and type of the underlying object and then contains the object.

A four byte magic number prefix:

Bytes 0-3: "k8s\x00" [0x6b, 0x38, 0x73, 0x00]

An encoded Protobuf message with the following IDL:

message Unknown {
  // typeMeta should have the string values for "kind" and "apiVersion" as set on the JSON object
  optional TypeMeta typeMeta = 1;

  // raw will hold the complete serialized object in protobuf. See the protobuf definitions in the client libraries for a given kind.
  optional bytes raw = 2;

  // contentEncoding is encoding used for the raw data. Unspecified means no encoding.
  optional string contentEncoding = 3;

  // contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf and is usually
  // omitted.
  optional string contentType = 4;
}

message TypeMeta {
  // apiVersion is the group/version for this type
  optional string apiVersion = 1;
  // kind is the name of the object schema. A protobuf definition should exist for this object.
  optional string kind = 2;
}

Generated Rust:

k8s-pb/k8s-pb-codegen/out/apimachinery.pkg.runtime.rs

Lines 87 to 104 in 7b4bf26

 #[derive(Clone, PartialEq, ::prost::Message)] 

 pub struct Unknown { 

 #[prost(message, optional, tag="1")] 

 pub type_meta: ::core::option::Option<TypeMeta>, 

 /// Raw will hold the complete serialized object which couldn't be matched 

 /// with a registered type. Most likely, nothing should be done with this 

 /// except for passing it through the system. 

 #[prost(bytes="vec", optional, tag="2")] 

 pub raw: ::core::option::Option<::prost::alloc::vec::Vec<u8>>, 

 /// ContentEncoding is encoding used to encode 'Raw' data. 

 /// Unspecified means no encoding. 

 #[prost(string, optional, tag="3")] 

 pub content_encoding: ::core::option::Option<::prost::alloc::string::String>, 

 /// ContentType is serialization method used to serialize 'Raw'. 

 /// Unspecified means ContentTypeJSON. 

 #[prost(string, optional, tag="4")] 

 pub content_type: ::core::option::Option<::prost::alloc::string::String>, 

 }