obmarg / kazan

Kubernetes API client for Elixir

Kazan

Kazan is a Kubernetes API client for Elixir. It uses the OpenAPI specifications provided by kube to generate most of it's functions and datastructures. This allows the whole kube API to be supported with relatively little effort.

Kazan should mostly work though it's not thoroughly tested against the actual kubernetes API. If you find a bug in the library please file an issue (or submit a PR) and I'll try and get it fixed.

I'm reasonably happy with the API at the moment so I don't expect to change it too drastically. However I can't speak for the k8s API specifications that the kazan code is generated from. Also the library is still pre-1.0 so if I find a better way to put it together I may end up changing things.

Looking for some help? Check out kazan's Gitter chatroom.

Features

• Support for most Kubernetes API calls.
• Structs for most Kubernetes API structures.
• Documentation of all models & calls.
• Client certificate, token and auth provider based authentication.
• Loading config from kubeconfig files.
• Support for watch requests.
• Typespecs for functions and structs (though dialyzer outputs a lot of warnings when run on Kazan)
• Limited support for custom resources. See the Kazan.Model documentation for more details.

Not Implemented

• Other forms of authentication
• Patching with application/json-patch+json or application/strategic-merge-patch+json content types.
• Removing fields when patching with application/merge-patch+json.
• Comprehensive tests.
• Validation of requests.
• Probably some other things.

Installation

If available in Hex, the package can be installed as:

1. Add kazan to your list of dependencies in mix.exs:

def deps do
  [{:kazan, "~> 0.11"}]
end

2. Ensure kazan is started before your application:

def application do
  [applications: [:kazan]]
end

Configuration

Kazan uses the Kazan.Server struct to contain server & authentication configuration details. Kazan.Server also provides some convenience functions to create a Kazan.Server from external sources such as a kube config file, or a kube service account.

If your application is only going to be talking to a single kubernetes cluster, then it's recommended to configure that cluster in your mix config. This makes working with kazan slightly easier, as you can call Kazan.run without providing a server every time.

In Cluster Authentication

If your code is going to be running on a kubernetes cluster and you wish to use the kubernetes service account that can be configured like this:

config :kazan, :server, :in_cluster

Alternatively, the Kazan.Server.in_cluster function can be used to create a server for passing straight into Kazan.run

Configuration via kube config file.

If you have a kube config file that contains the cluster & auth details you wish to use, kazan can use that:

config :kazan, :server, {:kubeconfig, "path/to/file"}

Alternatively, the Kazan.Server.from_kubeconfig function can be used to create a server for passing straight into Kazan.run

Configuring server details directly

If you wish to configure the server details manually, kazan can also accept a map of server parameters:

config :kazan, :server, %{url: "kubernetes.default" auth: %{token: "your_token"}}

See the Kazan.Server documentation to see what fields this supports.

Google Kubernetes Engine Config via gcloud

If developing against GKE, gcloud can create a kube config file that Kazan can understand. However, in this case you will need to call Kazan.Server.resolve_auth/2 in order to query gcloud for a valid token. See the docs for Kazan.Server.resolve_auth/2 for more details.

Usage

Making a request with Kazan is done in two stages.

1. Build the request object using one of the functions in Kazan.Api.*.
2. Run that request using Kazan.run.

For example, to get all of the pods from the server configured in the mix config:

Kazan.Apis.Core.V1.list_pod_for_all_namespaces!()
|> Kazan.run!()
# %Kazan.Models.V1.PodList{...}

Alternatively, you might want to specify the server to send the request to:

server = Kazan.Server.in_cluster()

Kazan.Apis.Core.V1.list_pod_for_all_namespaces!()
|> Kazan.run!(server: server)
# %Kazan.Apis.Core.V1.PodList{...}

More details on building requests can be found in the documentation for Kazan.Apis.

Details on creating watch requests can be found in the documentation for Kazan.Watcher.

kazan's Issues

Autolink module/api names in documentation

An excerpt from Kazan.Apis.Apps.V1beta1.ControllerRevision:

DEPRECATED - This group version of ControllerRevision is deprecated by apps/v1beta2/ControllerRevision. See the release notes for more information. ControllerRevision

In this sample, it'd be ideal if ControllerRevision was quoted, and if apps/v1beta2/ControllerRevision was converted into Kazan.Apis.Apps.V1beta2.ControllerRevision so that ExDoc would automatically link to the replacement module.

Not sure how hard this would be, but it seems like it'd be worth it.

It might also be possible for some names of modules that aren't fully scoped to somehow become links. Though this might need some work done in ExDoc. As repeating the full module path every time in a sentence is going to become tiresome.

Cannot read config

iex(3)> server = Kazan.Server.from_kubeconfig(System.user_home <> "/.kube/config")
** (FunctionClauseError) no function clause matching in Kazan.Server.get_cert/2    
    
    The following arguments were given to Kazan.Server.get_cert/2:
    
        # 1
        %{"insecure-skip-tls-verify" => true, "server" => "https://localhost:6443"}
    
        # 2 
        "/Users/b/.kube"
    
    Attempted function clauses (showing 2 out of 2):
    
        defp get_cert(%{"certificate-authority" => certfile}, basepath)
        defp get_cert(%{"certificate-authority-data" => certdata}, _)
    
    (kazan) lib/kazan/server.ex:255: Kazan.Server.get_cert/2
    (kazan) lib/kazan/server.ex:57: Kazan.Server.from_kubeconfig/2

Not compiling on Elixir 1.7-dev

==> kazan
Compiling 16 files (.ex)
warning: variable "other" is unused
  lib/kazan/server.ex:186


== Compilation error in file lib/kazan/models.ex ==
** (UndefinedFunctionError) function String.Casing.titlecase_once/1 is undefined or private. Did you mean one of:

      * titlecase_once/2

    (elixir) String.Casing.titlecase_once("apis")
    lib/kazan/codegen/models.ex:58: anonymous fn/1 in Kazan.Codegen.Models.module_name/2
    (elixir) lib/enum.ex:1296: Enum."-map/2-lists^map/1-0-"/2
    lib/kazan/codegen/models.ex:55: Kazan.Codegen.Models.module_name/2
    (elixir) lib/enum.ex:745: anonymous fn/3 in Enum.each/2
    (stdlib) lists.erl:1263: :lists.foldl/3
    (elixir) lib/enum.ex:1917: Enum.each/2
    lib/kazan/codegen/models.ex:101: Kazan.Codegen.Models.parse_models/1
could not compile dependency :kazan, "mix compile" failed. You can recompile this dependency with "mix deps.compile kazan", update it with "mix deps.update kazan" or clean it with "mix deps.clean kazan"

Move models under their corresponding API modules.

Most of the models in Kazan correspond with a particular API & API version. For example Kazan.Apis.RbacAuthorizationV1.list_role_for_all_namespaces returns Kazan.Models.Rbac.V1.RoleList.

I'm thinking it would make more sense if models were grouped with the APIs that they correspond with. For the above example this would mean Kazan.Apis.RbacAuthorization.V1.list_role_for_all_namespaces would return a Kazan.Apis.RbacAuthorization.V1.RoleList.

This would make grouping documents as suggested in #20 a bit easier - the groups could correspond with the individual APIs.

Integration test improvement suggestions

I suggest the following changes to the integration test:

1. Create a namespace and run tests that create objects against that namespace - this allows for the namespace to be deleted in a module levelon_exit callback, thus ensuring everything is cleaned up.
2. Use the alpine image for the pod that is created rather than obmarg/health-proxy as alpine is small and will be pulled quickly.
3. Do not assume there are no ClusterRoles in the test RBAC Authorization V1 Beta 1 API as this depends on having a fresh K8S installation.

handling Kubernetes Errors

The StatusError that is returned when a call from kubernetes doesn't succeed would be useful.

Currently its just being decoded https://github.com/obmarg/kazan/blob/master/lib/kazan/client/imp.ex#L160

New `Kazan.Models` Module Orgnanization for `ApiMachinery`

So I'm working with kazan's master branch which has the new module organization changes. Based on the comment in the commit, I assume that the ObjectMeta would still be under the same location as it is now.

However, when I alias Kazan.Models.ApiMachinery.Apis.Meta.V1.ObjectMeta, I get this error message:

== Compilation error in file lib/ephemeral/kube.ex ==
** (CompileError) lib/ephemeral/kube.ex:28: Kazan.Models.ApiMachinery.Apis.Meta.V1.ObjectMeta.__struct__/1 is undefined, cannot expand struct Kazan.Models.ApiMachinery.Apis.Meta.V1.ObjectMeta
    (stdlib) lists.erl:1354: :lists.mapfoldl/3
    (stdlib) lists.erl:1355: :lists.mapfoldl/3
    (elixir) expanding macro: Kernel.@/1
    lib/ephemeral/kube.ex:28: Ephemeral.Kube (module)

I'm not sure where it is now in master branch so it would be nice if there was a little more description for models that are staying in the same module structure or if they changed in some other way unlike the other API models.

Thank you,

Patch requests are not properly suppported

Kube expects PATCH requests to use application/merge-patch+json or application/json-patch+json syntax. Currently we're just sending the whole body as we do with POST requests.

Will need to figure out how to fix this...

Error handling strategy

I am looking for some ideas on handling errors in a user-friendly manner. My main requirement is to distinguish transient errors (which may be recoverable) and hard errors (which require user action). Here is an example. I was watching a k8s cluster and the secrets were changed. My code got a bad_match error (shown below) at https://github.com/obmarg/kazan/blob/master/lib/kazan/watcher.ex#L135.

{:error,
{{:badmatch,
{:error,
{:http_error, 401,
%{
"apiVersion" => "v1",
"code" => 401,
"kind" => "Status",
"message" => "Unauthorized",
"metadata" => %{},
"reason" => "Unauthorized",
"status" => "Failure"
}}}},
[
{Kazan.Watcher, :init, 1, [file: 'lib/kazan/watcher.ex', line: 135]},
{:gen_server, :init_it, 2, [file: 'gen_server.erl', line: 374]},
{:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 342]},
{:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 249]}
]}}

I need to get at the root cause of the error to know if this is a recoverable error or not. I would prefer to not unwrap the error which would be a brittle way to handle this. Could this be {Ok, _} or {Err, _} response instead?

@obmarg Any thoughts on this?

Compilation error with Elixir 1.14.1

Environment

Elixir & Erlang/OTP versions (elixir --version):

Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]
Elixir 1.14.1 (compiled with Erlang/OTP 25)

Which version of Kazan are you using? (cat mix.lock | grep kazan):

"kazan": {:hex, :kazan, "0.11.0", "fe70a92a922f307680086ecbfda085f11193649dd84b465e488d5b82f96e7f7b", [:mix], [{:httpoison, "~> 0.10 or ~> 1.0", [hex: :httpoison, repo: "hexpm", optional: false]}, {:poison, "~> 2.0 or ~> 3.0 or ~> 4.0", [hex: :poison, repo: "hexpm", optional: false]}, {:yaml_elixir, "~> 2.0", [hex: :yaml_elixir, repo: "hexpm", optional: false]}], "hexpm", "8786019fb0af3cef1e9d2a322ce4f12d1e03740345955a15f85d39dac03a97f0"},

Current behavior

To demonstrate this behavior I created an application through the command mix new dummy_app and added kazan as a dependency. Once that I try to compile the application through the command mix compile I get the error:

== Compilation error in file lib/kazan/server/certificate_auth.ex ==
** (CompileError) lib/kazan/server/certificate_auth.ex:9: cannot define module Inspect.Kernel because it is currently being defined in lib/kazan/server/basic_auth.ex:8
    lib/kazan/server/certificate_auth.ex:9: (module)
could not compile dependency :kazan, "mix compile" failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile kazan", update it with "mix deps.update kazan" or clean it with "mix deps.clean kazan"

Expected behavior

If I change to using Elixir 1.14.0 instead of 1.14.1 I can compile successfully.

Elixir & Erlang/OTP versions (elixir --version):

Erlang/OTP 25 [erts-13.0.4] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]
Elixir 1.14.0 (compiled with Erlang/OTP 25)

Generated kazan app
==> dummy_app
Compiling 1 file (.ex)
Generated dummy_app app

Property documentation renders a bit odd

Parameter documentation that has line breaks ends up rendering weird, because it breaks the bulleted list we're in the middle of. Would be good to fix this somehow? Possibly convert paragraphs to bullet entries?

An example of the issue:

Group pages in API documentation

The ecto documentation groups related modules under headers, like this:

The k8s API is pretty big, so kazan has a ton of modules. It'd be good to think about if the kazan docs could be grouped in a similar manner to the ecto docs.

Doctests

Some public facing kazan functions could do with examples in their documentation. Ideally these should be in the form of doc tests so we can be sure they're correct.

This might involve writing a mock-client, mock k8s or using exvcr so that the tests can run without an actual k8s server.

Format code with Elixir 1.6 formatter

I suggest that to improve consistency, the code is formatted with the Elixir 1.6 formatter, a .formatter.exs is committed, and the Travis configuration is changed to run mix format --check-formatted in addition to the default mix test.

Happy to create a PR for this.

Consider utility functions for setting names etc.

Naming objects you create with kazan is not ideal right now - names live inside ObjectMeta, so you need to do something like:

alias Kazan.Apis.Core.V1, as: Core

%Core.Namespace{
    metadata: %Kazan.Models.Apimachinery.Meta.V1.ObjectMeta{name: "test"}
}
|> Core.create_namespace!() 
|> Kazan.run!

It might be worth exploring a pipe function that sets the name/other metadata:

%Core.Namespace{}
|> with_name("test")
|> Core.create_namespace!() 
|> Kazan.run!

or

%Core.Namespace{}
|> with_meta(name: "test")
|> Core.create_namespace!() 
|> Kazan.run!

It's not clear what module these functions could live in. But it's worth thinking about.

New features need integration tested

There's been a few new features added lately that'd be good to integration test:

• In config file certificate auth
• Token auth
• In cluster auth

Regarding custom resources

I'm trying to integrate CRD:s into my app. I know Kazan does not suport CRD yet, but I think it would be a step in the right direction if it would be possible to use Kazan.run with a custom Kazan.Request.response_scheme in a similar way to how nested structs can be decoded using Poison?

  def list_namespaced_healthchecks!(namespace, -opts) do
    %Kazan.Request{
      body: nil,
      method: "get",
      path: "/apis/group/v1alpha1/namespaces/#{namespace}/healthchecks",
      query_params: %{},
      response_schema: %HealthcheckList{items: [%Healthcheck{}]}
    }
  end

or maybe something simpler like run_raw to skip the decoding step in run. In my opinion is no problem creating custom functions returning a %Kazan.Request{} for my CRD and doing encode/decode manually. It would be nice to have the option to do so, since right now I sort of have to duplicate the Kazan.run and Kazan.Watcher logic since I can't work around it.

Support alternative swagger.json file

We need to support the K8S metrics-server API (https://github.com/kubernetes-incubator/metrics-server) in our implementation.

To do this I need to fork the Kazan repo to make the following changes:

• Use a different swagger.json which included the metrics-server endpoints
• Add two lines to Kazan.Codegen.Naming to map the metrics namespace

(chazsconi@cd923bc#diff-1856cab08f35cbaee3b304822e1699c4)

It would be nice if these two things could be set via the configuration.

Additionally I also had to manually change the swagger.json so the window field is a string rather than a Duration cause a crash when decoding, but this is really a different issue, and I'm not sure how to fix it properly.

chazsconi@0e44c4e

Required properties on models in k8s spec are ignored

For API definitions we use the required property in the k8s spec to decide where the argument should go.

Models also have required properties, but we don't do anything with them. We should probably do something with them. At the very least update our documentation to show which properties are required. Perhaps even enforce them with @enforce_keys, though this could make building models more awkward so may not be worth it...

Use x-kubernetes properties to backlink from Model -> Api function

The k8s specs contain x-kubernetes-group-version-kind and x-kubernetes-action fields for some models & api actions. We could use these to back-link from models to the functions that take/return the models. Worth considering.

Write a CONTRIBUTING document.

I need to write some details on how to contribute to the project.

Default file name for ca_cert in `Kazan.Server.in_cluster`

I notice the in_cluster function is looking for ca.key file name.

I believe this have changed according to Kubernetes' documentation on accessing the API server from a pod

If available, a certificate bundle is placed into the filesystem tree of each container at
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,
and should be used to verify the serving certificate of the apiserver.

I am running into this error while testing.

** (File.Error) could not read file "/var/run/secrets/kubernetes.io/serviceaccount/ca.key": no such file or directory
    (elixir) lib/file.ex:272: File.read!/1
    (kazan) lib/kazan/server.ex:130: Kazan.Server.cert_from_pem/2
    (kazan) lib/kazan/server.ex:70: Kazan.Server.in_cluster/1
    (ephemeral) lib/ephemeral/kube.ex:154: Ephemeral.Kube.deploy/2

The documentation is from version 1.9 and I'm running 1.8 so at the very least these 2 versions are not using ca.key name anymore.

If my hypothesis is right, please let me know. I am happy to make a PR as well.

Thank you,

Release a new version

I'm currently using {:yaml_elixir, "~> 2.0"} in my mix project, so I have to use {:kazan, git: "https://github.com/sambooo/kazan.git", branch: "master"} for dependency resolution to work. Please do a new release so I can remove this constraint.

K8s api implementations are now significantly out of date

Most notable of these are networking.k8s.io/v1 implementation of Ingress. But I'm pretty sure there are a large number of other deprecated resources in the set provided by kazan alongside other resources missing.

Better support for specifying server in Mix config

We currently support (and mention in the README) configuring a kubernetes server in the mix config.

However, the Kazan.Server functions (in_cluster and from_kubeconfig) are not available to call from the mix config. It'd be good to provide an alternative way of specifying these in the mix config.

Watcher process leaks if using old version of HTTPoison

Due to edgurgel/httpoison#375 (which I created a PR to fix, and is now merged) if you use HTTPoison < v1.6.1 you will have a process leak when using watches whenever a GONE event is received from K8S.

Therefore should Kazan's deps be changed to force using this version or later of HTTPoison, or should this just be warned about in the docs?

I'm happy to create a PR for either option.

Cut down on `String.to_atom` uses

There's a bunch of places in the codebase that call String.to_atom. Sometimes this will be unavoidable, as we actually need to create new atoms . However, should look into whether they're always safe & needed. Similar to the fix in 87e7e21

Allow replacing of Kazan.Client.Imp via configuration

As mentioned in #72 we have a wrapper around Kazan.run/2 for instrumentation.

It would be useful if the module that Kazan.run/1 and Kazan.run/2 delegates to could be changed by configuration. This way we could easy provide our own implementation which wraps calls to Kazan.Client.Imp.run/2 with instrumentation.

This would also be useful to allow Kazan.Client.Imp to be replaced by a mock module for testing an app that uses Kazan.

To do this I would also suggest moving Kazan.Client.Imp.run!/2 to the Kazan module (as it's just a wrapper around run/2) and leaving run/2 as the only function that is required to be implemented in Kazan.Client.Imp or a mock implementation. A behaviour could also be added.

@obmarg I can provide a PR for this if you agree with the approach.

Add auth config documentation

I'm trying to figure out how to configure the package, especially in the area of authenticating through ServiceAccount. I saw some the code in Kazan.Server on reading the kubecfg file. However, I'm not really sure where it's used, and the documentation isn't saying much about how to setup the auth component.

I'll keep looking around the repo but it would be really nice to have more info/example in the documentation or README.

I'm also happy to submit PR if you'd preferred.

Support Watch requests

@obmarg In my project I need to support watch requests.

To do this I have written some code that can be used in the client as follows:

1. You create a request in the normal way, but setting watch=true e.g.

request = Kazan.Apis.CoreV1.list_namespace!(watch: "true")

2. You start a watcher using the request, and passing the initial resource version and a pid to which to send received messages.

Kazan.Watcher.start_link(request, resource_version: rv, send_to: self())

3. In your client code you receive messages for each %Versioned.Event{}. You can pattern match on the object type if you have multiple watchers configured. For example, if the client is a GenServer

  # type is "ADDED", "DELETED" or "MODIFIED"
  def handle_info(%Kazan.Models.Versioned.Event{object: object, type: type}, state) do
    case object do
      %Kazan.Models.V1.Namespace{} = namespace ->
        process_namespace_event(type, namespace)

      %Kazan.Models.V1.Job{} = job ->
        process_job_event(type, job)
    end
    {:noreply, state}
  end

Internally the Watcher does the following:

1. Initiates the request to HTTPoison using an AsyncRequest
2. Buffers chunks received from HTTPoison until a complete line of JSON is received.
3. Decodes the JSON and then decodes the K8S objects using Kazan.Models.decode/1
4. Sends the event to the client
5. Keeps track of the latest resource_version object received from K8S
6. Triggers another request, passing the latest resource_version when a HTTPoison timeout is received.

What do you think of this approach? If you think this is the right way to do it, I will tidy up the code and submit a PR.

If the response is not JSON then it should not be decoded

Some requests, e.g: Core.read_namespaced_pod_log/3 return a response type of text/plain and not application/json

In this case the response should not be JSON decoded and should just be returned as the raw response.

I'll create a PR to fix this by checking the response type before JSON and Model decoding.

Provide convenience functions for CRUD of custom resources

#56 added custom resource support to kazan. However, CRUD requests on custom resources require a user to manually build a Kazan.Request type. It'd be good to provide some utility functions for building these requests.

I'm thinking something along the lines of:

defmodule Kazan.CustomResources do
  @spec create(struct) :: Kazan.Request
  @spec delete(struct) :: Kazan.Request
  # Etc.
end

Watcher fails with 410 "GONE"

If one particular event type is been watched (e.g. Namespace changes) and no new events happen for a long time, but other events, that are not being watched happen, the other, non-watched events, cause the RV to increase.

Eventually the RV that is stored in the watcher is too old to be used, and the watcher fails returning a 410 response code.

If an original RV was passed to the watcher, the supervisor restarts the watcher with the same init parameters, and it fails again.

Possible solutions:

1. Parse the message that comes back to extract a later RV

2. Send a message to the caller to inform it that it can no longer watch and it should start again

3. Somehow obtain the latest RV from K8S without potentially missing events

Logs below:

FunctionClauseError) no function clause matching in Kazan.Watcher.extract_rv/1
    (kazan) lib/kazan/watcher.ex:235: Kazan.Watcher.extract_rv(%{"apiVersion" => "v1", "code" => 410, "kind" => "Status", "message" => "too old resource version: 20293058 (20295457)", "metadata" => %{}, "reason" => "Gone", "status" => "Failure"})
    (kazan) lib/kazan/watcher.ex:219: anonymous fn/4 in Kazan.Watcher.process_lines/2
    (elixir) lib/enum.ex:1899: Enum."-reduce/3-lists^foldl/2-0-"/3
    (kazan) lib/kazan/watcher.ex:158: Kazan.Watcher.handle_info/2
    (stdlib) gen_server.erl:616: :gen_server.try_dispatch/4
    (stdlib) gen_server.erl:686: :gen_server.handle_msg/6
    (stdlib) proc_lib.erl:247: :proc_lib.init_p_do_apply/3
Last message: %HTTPoison.AsyncChunk{chunk: "{\"type\":\"ERROR\",\"object\":{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"too old resource version: 20293058 (20295457)\",\"reason\":\"Gone\",\"code\":410}}\n", id: #Reference<0.4253789252.1463549953.187567>}

Expose function to pull config from application config

Adding the Kazan.Server.resolve_token function has meant that people using GCP can no longer use the kazan application config. It would be good to expose a function to construct a Kazan.Server from any application config to handle this particular usecase.

Expose context in %Kazan.Server

I'm currently playing around with Kazan and I find it very pleasant to use, thank you for working on it!

I find myself switching between two contexts at the moment, and realised that it would be nice to have context exposed on the server to see that the current server I'm working with is in the correct context.

Generate typespecs

I reckon it'd be nice to be able to check the validity of struct-modifying code with Dialyzer. Given the types are already present in the docs it seems the required information is there to do this.

Getting Kubernetes Logs

I'd love to use this library to stream from the kubernetes logs endpoint. This api is however, not in the swagger documentation.

Watching CRDs is not supported

#65 added support for decoding & encoding custom resources - when a Kazan.Request is built with a custom resource as it's response_model we'll succesfully decode that custom resource.

Watch request decoding however uses the kind field in the object in the WatchEvent. The lookup table for kinds is generated at compile time, so custom resources will not be included in this. We probably need to update the lookup table to be dynamic somehow.