nginxinc / nginx-loadbalancer-kubernetes

A Kubernetes Controller to synchronize NGINX+ Resources with Kubernetes Ingress Resources

nginx-loadbalancer-kubernetes

The NGINX Loadbalancer for Kubernetes, or NLK, is a Kubernetes controller that provides TCP load balancing external to a Kubernetes cluster running on-premise.

Requirements

What you will need

• A Kubernetes cluster running on-premise.
• One or more NGINX Plus hosts running outside your Kubernetes cluster (NGINX Plus hosts must have the ability to route traffic to the cluster).

There is a more detailed Installation Reference available in the docs/ directory.

Why NLK?

NLK provides a simple, easy-to-manage way to automate load balancing for your Kubernetes applications by leveraging NGINX Plus hosts running outside your cluster.

NLK installs easily, has a small footprint, and is easy to configure and manage.

NLK does not require learning a custom object model, you only have to understand NGINX configuration to get the most out of this solution. There is thorough documentation available with the specifics in the docs/ directory.

What does NLK do?

tl;dr:

NLK is a Kubernetes controller that monitors Services and Nodes in your cluster, and then sends API calls to an external NGINX Plus server to manage NGINX Plus Upstream servers automatically.

That's all well and good, but what does it mean? Kubernetes clusters require some tooling to handling routing traffic from the outside world (e.g.: the Internet, corporate network, etc.) to the cluster. This is typically done with a load balancer. The load balancer is responsible for routing traffic to the appropriate worker node which then forwards the traffic to the appropriate Service / Pod.

If you are using a hosted Kubernetes solution -- Digital Ocean, AWS, Azure, etc. -- you can use the cloud provider's load balancer service. Those services will create a load balancer for you. You can use the cloud provider's API to manage the load balancer, or you can use the cloud provider's web console.

If you are running Kubernetes on-premise and will need to manage your own load balancer, NLK can help.

NLK itself does not perform load balancing. Rather, NLK allows you to manage Service resources within your cluster to update your load balancers, with tooling you are most likely already using.

Getting Started

There are few bits of administrivia to get out of the way before you can start leveraging NLK for your load balancing needs.

As noted above, NLK is intended for when you have one or more Kubernetes clusters running on-premise. In addition to this, you need to have at least one NGINX Plus host running outside your cluster (Please refer to the Roadmap for information about other load balancer servers).

Deployment

RBAC

As with everything Kubernetes, NLK requires RBAC permissions to function properly. The necessary resources are defined in the various YAML files in deployment/rbac/.

For convenience, two scripts are included, apply.sh, and unapply.sh. These scripts will apply or remove the RBAC resources, respectively.

The permissions required by NLK are modest. NLK requires the ability to read Resources via shared informers; the resources are Services, Nodes, and ConfigMaps. The Services and ConfigMap are restricted to a specific namespace (default: "nlk"). The Nodes resource is cluster-wide.

Configuration

NLK is configured via a ConfigMap, the default settings are found in deployment/configmap.yaml. Presently there is a single configuration value exposed in the ConfigMap, nginx-hosts. This contains a comma-separated list of NGINX Plus hosts that NLK will maintain.

You will need to update this ConfigMap to reflect the NGINX Plus hosts you wish to manage.

If you were to deploy the ConfigMap and start NLK without updating the nginx-hosts value, don't fear; the ConfigMap resource is monitored for changes and NLK will update the NGINX Plus hosts accordingly when the resource is changed, no restart required.

There is an extensive Installation Reference available in the docs/ directory. Please refer to that for detailed instructions on how to deploy NLK and run a demo application.

Versioning

Versioning is a work in progress. The CI/CD pipeline is being developed and will be used to build and publish NLK images to the Container Registry. Once in place, semantic versioning will be used for published images.

Deployment Steps

To get NLK up and running in ten steps or fewer, follow these instructions (NOTE, all the aforementioned prerequisites must be met for this to work). There is a much more detailed Installation Reference available in the docs/ directory.

1. Clone this repo (optional, you can simply copy the deployments/ directory)

git clone [email protected]:nginxinc/nginx-loadbalancer-kubernetes.git

2. Apply the Namespace

kubectl apply -f deployments/deployment/namespace.yaml

3. Apply the RBAC resources

./deployments/rbac/apply.sh

4. Update / Apply the ConfigMap (For best results update the nginx-hosts values first)

kubectl apply -f deployments/deployment/configmap.yaml

5. Apply the Deployment

kubectl apply -f deployments/deployment/deployment.yaml

6. Check the logs

kubectl -n nlk logs -f $(kubectl -n nlk get po -l "app=nlk" --no-headers -o custom-columns=":metadata.name")

At this point NLK should be up and running. Now would be a great time to go over to the Installation Reference and follow the instructions to deploy a demo application.

Monitoring

Presently NLK includes a fair amount of logging. This is intended to be used for debugging purposes. There are plans to add more robust monitoring and alerting in the future.

As a rule, we support the use of OpenTelemetry for observability, and we will be adding support in the near future.

Contributing

Presently we are not accepting pull requests. However, we welcome your feedback and suggestions. Please open an issue to let us know what you think!

One way to contribute is to help us test NLK. We are looking for people to test NLK in a variety of environments.

If you are curious about the implementation, you should certainly browse the code, but first you might wish to refer to the design document. Some of the design decisions are explained there.

Roadmap

While NLK was initially written specifically for NGINX Plus, we recognize there are other load-balancers that can be supported.

To this end, NLK has been architected to be extensible to support other "Border Servers". Border Servers are the term NLK uses to describe load-balancers, reverse proxies, etc. that run outside the cluster and handle routing outside traffic to your cluster.

While we have identified a few potential targets, we are open to suggestions. Please open an issue to share your thoughts on potential implementations.

We look forward to building a community around NLK and value all feedback and suggestions. Varying perspectives and embracing diverse ideas will be key to NLK becoming a solution that is useful to the community. We will consider it a success when we are able to accept pull requests from the community.

Authors

• Chris Akker - Solutions Architect - Community and Alliances @ F5, Inc.
• Steve Wagner - Solutions Architect - Community and Alliances @ F5, Inc.

License

Apache License, Version 2.0

© F5, Inc. 2023

(but don't let that scare you, we're really nice people...)

nginx-loadbalancer-kubernetes's Issues

Ensure Informer is not nil in initializeEventHandlers Method

In the Run method, there's a check for nil informer but no such check in initializeEventHandlers method which also accesses the informer. It would be prudent to add a nil check for the informer in the initializeEventHandlers method to avoid potential nil pointer dereference issues.

Document code files

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is. Ex. I'm always frustrated when ...

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Sign Docker images

Is your feature request related to a problem? No problem, no.

Describe the solution you'd like

When an image is built, sign it before adding to the repo. Check out Content trust in Docker for more details.

Additional context

This will be important for broader adoption among companies that have elevated security profiles.

Use obj2 for Updated Object in handleUpdateEvent Method

Describe the bug

The Certificates::handleUpdateEvent method uses the first argument to update the value of the Certificate. The first argument represents the old value, the second argument is the new value.

This bug will cause Certificate rotation to fail.

To reproduce

With a running nginx-loadbalancer-kubernetes, update one of the Secrets containing a Certificate. Updating the NGINX Plus host(s) may fail, depending on how aggressively certificate management is implemented.

Expected behavior

The internal state should be updated with the new certificate value.

Additional context

In the handleUpdateEvent method, it should use obj2 instead of obj to get the updated value of the object to reflect the latest changes.

Support HA NGINX+ servers

Is your feature request related to a problem? Please describe

For customers that wish to have NGINX+ LoadBalancers (LBs) on the edge in a High Availability (HA) configuration, the controller needs to support updating each NGINX+ instance individually.

Describe the solution you'd like

One that works. One that is not overly complicated. One that is maintainable.

Describe alternatives you've considered

• Use the same NGINX+ client for multiple instances
• Use an NGINX+ client for each instance

Additional context

This is still a Prototype / Proof-of-Concept implementation, but care should still be taken to ensure a smooth transition to production-quality code.

Fix code scanning alert -

TLS considerations in client

Tracking issue for:

• https://github.com/nginxinc/nginx-k8s-loadbalancer/security/code-scanning/18

Add health endpoints

Is your feature request related to a problem? Please describe

The problem is that Kubernetes currently cannot monitor the controller's state and act ensure that the controller is running.

Describe the solution you'd like

Follow the Kubernetes API health endpoints guidance to implement /livez and /readyz endpoints.

A clear and concise description of what you want to happen.

Describe alternatives you've considered

None, this is a standard way of handling this.

Additional context

N/A

Use NGINX terms for Client type selection

Is your feature request related to a problem? Please describe

No problem.

To be consistent with the naming the NGINX naming should be used to not confuse users. Instead of "tcp" and "http", use "stream" and "http".

Describe the solution you'd like

I think I just did.

Describe alternatives you've considered

The current naming, which may cause confusion.

Additional context

I got nothing.

Allow selection of L4 / L7 Upstream updates

Is your feature request related to a problem?

No

Describe the solution you'd like

Need the ability to target either HTTP or TCP Upstreams in the NGINX Plus edge server.

Describe alternatives you've considered

• Change support from TCP to HTTP and not allow selection.

Additional context

Fail fast on this one. Time-box to one day and short circuit to supporting only HTTP for the time being.

Routing question / VXLAN

Hello,

More a question than a feature request.

Does this work without ibgp peering / calico / pods having routable IPs from outside of the cluster ?

and if not, are there any plans to introduce a model similar to Big IP Controller where traffic is tunneled into the cluster from the external NGINX load balancer over VXLAN (or some other layer 2/3 tunnel) ?

Thanks.

Add SBOM

Is your feature request related to a problem? No

Describe the solution you'd like

A Software Bill Of Materials (SBOM) should be generated for builds. Perhaps the Kubernetes tools could be of use.

Additional context

SBOMs are increasing in importance for many clients; this project should include the artifact for greater acceptance.

Move the nginx client creation into the TcpBorderClient and HttpBorderClient

Is your feature request related to a problem? Please describe

No problem.

Synchronizer should not know about the NGINX Plus client. It should simply provide an HttpClient to the NewBorderClient factory method.

Describe the solution you'd like

In order to facilitate extensibility for various Border Server types, no assumptions should be made about which underlying client is used. The BorderClient implementation should contain that knowledge (e.g.: The HttpBorderClient and TcpBorderClient each know they need the NGINX Plus client, they can create those using the given HttpClient).

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Can `settings.Run()` (line 47) be moved with the other Go routine startups?

the go settings.Run() call is done fairly early in main(). The original thought was this would give the Go routine time to load the certificates before the rest of the controller bits got up and running (sketchy trying to out-clever async code!).

However, the code was restructured to not wait for the inevitable callback from the Kubernetes API on startup, but rather to Read the targets before starting the listeners. This avoids a race condition and -- possibly? -- means the startup can be moved closer to the others.

Context: cmd/nginx-loadbalancer-kubernetes/main.go:47 go settings.Run()

SECURITY: Make the Certificates an unloggable type

Describe the bug

Leaking certs in logs is not acceptable. To help ensure this doesn't happen make the Certificates type (internal/certification/certificates.go) unloggable as described here: https://www.commonfate.io/blog/prevent-logging-secrets-in-go-by-using-custom-types

`no-tls` should be an explicitly handled case

Sauce: #120 (comment)

If someone were to typo the TLS Mode in the config, we may wish to fail startup. Couple of thoughts here:

• If the choice is to fail startup, how to best raise an alert that can be seen? Is it better to start in insecure mode (no-tls) or possibly fail to start and have it not be noticed?
• As an implementation detail (hate to be too prescriptive, but in this case I'm going to be), the TLS Modes should be iota-based values, NOT STRINGS. This will help avoid a whole class of issues.

Create a "Getting Started" document

Is your feature request related to a problem? Please describe

A clear and concise description of what the problem is. Ex. I'm always frustrated when ...

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

NLK Basic Auth Support

Is your feature request related to a problem? Please describe

The /api endpoint on the server I set up with Nginx Plus is protected with basic authentication. However, NLK cannot access this endpoint, and I believe this is an important feature.

Describe the solution you'd like

We can store a password for BasicAuth in the Secret. If the application finds this field filled, it should use it.

Additional context

When I examined the Nginx Plus Go Client, I noticed that such a structure does not exist, and it seems that development needs to be done there as well.

Question about ingress controller

Hello,

One question, can this project work with Ingress nginx or does it only function with Nginx Inc controller?

Thank you!

NLK is not working as expected

Describe the bug

NLK adds worker node machines as upstreams in the NGINX Plus servers.
I deployed the NGINX Ingress controller as a NodePort Service.
I have three worker machines, but only one pod for the NGINX Ingress controller. This means that two worker nodes do not have any NGINX Ingress controller pods.

When traffic arrives at a node that does not have any NGINX Ingress controller pod, it cannot reach the pods.

Nginx Dashboard displays upstreams as unhealthy.

I think;
#160
will solve the problem.

Is there anything specific I need to pay attention to?
The private IP addresses of the worker nodes and the Kubernetes CIDR block must match?

Expected behavior

NLK must add only nodes with nginx-ingress controller.

Your environment

Used helm chart.

image:
registry: ghcr.io
repository: nginxinc/nginx-loadbalancer-kubernetes
pullPolicy: Always
# Overrides the image tag whose default is the chart appVersion.
tag: latest

Bugs in Helm Chart + Harding coding

Describe the bug

helm template renders some errors.

To reproduce

Steps to reproduce the behavior:

1. helm template charts/nlk
2. See error

Expected behavior

helm chart not tested

Your environment

helm v3.12.x

Additional context

A Golden partner to Nginx Plus

Improve Error Handling in Event Handlers

In handleAddEvent, handleDeleteEvent, and handleUpdateEvent, there's a lack of error handling if the type assertion fails other than logging an error. Consider adding more graceful error handling in these scenarios.

[RFE] add NKL to project name and descriptions for SEO

NKL is the first thing I think of when I'm trying to remember this project and bring to others attention. Unfortunately, that abbreviation is not in any of the titles or main descriptions so it doesn't show up in a google search.

Optimizing these fields for Search Engine Optimization would really help users find the project.

Keep up the great work!

Genericize BorderClient creation in Synchronizer using an interface

Is your feature request related to a problem? Please describe

No Problem.

In order to allow extension and implement support for various Border Servers the Synchronizer module should not import the NGINX Plus client ("github.com/nginxinc/nginx-plus-go-client/client") directly. It does this presently in 'Synchronizer::buildBorderClient' method on line 93.

This direct creation of a concrete implementation should use an interface and factory to create an instance of a configurable implementation.

Describe the solution you'd like

Define an interface that is generic against the NGINX Plus client that can be extended for other clients down the road (sure, this is a modicum of 'future-proofing', but will hopefully incentivize more developers to join in the fun and contribute.

Describe alternatives you've considered

The current implementation is the alternative. Use clients directly with no separation that would allow for easier testing and extension.

Additional context

Add any other context or screenshots about the feature request here.

Research other logging libraries

Is your feature request related to a problem? Please describe

logrus is fine, but it's in gone into maintenance mode.

Look into the libraries mentioned in the README...

At a quick glance, Apex Log looks interesting.

Other options:

• Zap
• zerolog

It may be not worth changing logging, but it's good to be sure. Initial thoughts:

• Apex log's handlers look interesting; it might be worth providing the ability to direct the logs to the back end of choice for the user.
• While the use of log-levels was intentional, it might warrant a quick review.
• Does one of these libraries support a more deliberate and LOUD approach to misconfiguration, especially with the TLS configuration. (See the conversation from issue 143 for more context.

Describe the solution you'd like

Logging, done more right than it already is; which is pretty all right.

Describe alternatives you've considered

Alternatives are listed above.

Additional context

CI / CD with Github Actions

Is your feature request related to a problem? Please describe

Just what the title says.

Describe the solution you'd like

• Build, test, and deploy an image to the GitHub Container registry.

Describe alternatives you've considered

CircleCi and Docker Hub.

Docker Hub is apparently falling out of favor for being too old (old and busted vs the new hotness).

Additional context

This is important.

Make upstreams route only to nodes known to contain ingress pods

Is your feature request related to a problem? Please describe

Currently NLK populates the host/port of all nodes in the cluster to the NGINX upstream block. If a request goes to a node that does not have a pod running the NGINX Ingress controller, kube-proxy will take care of getting it there. However, it seems that it would be better to route the request to the ingress-containing node to begin with since we have that information.

Describe the solution you'd like

When we receive updates about a new configuration of the cluster/controller, NLK should compare the node list to the locations of the ingress pods, and only send those upstreams to NLK. That means upstreams should change when:

• We scale the ingress
• When the ingress is deployed for the first time
• When we add nodes to the cluster only if this creates changes to the ingress pods
• Ingress pods are rebalanced across nodes.

If there are no nodes containing ingress pods, perhaps it is best then to send all notes and dispatch an event about the trouble. That way at least we consistently send traffic to the cluster and don't drop it if we have some bug/race condition in figuring out ingress pod assignments.

Describe alternatives you've considered

The current operation works fine due to kube-proxy picking up the slack. I don't see any other ways of achieving this.

Additional context

Support namespaces

Is your feature request related to a problem? Please describe

I'm working on the helm charts for the NLK, found some limitations with the current NLK implementation.

Describe the solution you'd like

Support namespaces for both nginx-ingress and nlk.

Describe alternatives you've considered

An alternative solution is unavailable.

Initialize Certificates Map to Avoid Nil Map Panic

In the NewCertificates method, the Certificates field is initialized to nil, which could cause a panic when trying to insert key-value pairs into the map. It should be initialized to an empty map instead: Certificates: make(map[string]map[string][]byte).