nginxinc / nginx-openshift-router

NGINX and NGINX Plus OpenShift Routers

NGINX OpenShift Router

This repository has been archived. There will likely be no further development on the project and security vulnerabilities may be unaddressed.

This repo includes an implementation of an OpenShift Router based on NGINX and NGINX Plus.

What is the Router?

The Router provides edge load balancing for applications on OpenShift, acting as an entry point for all external requests coming to applications running on OpenShift. The Router’s job is to identify which application a particular request is sent to and then route it to an instance (pod) of that application. OpenShift includes a special resource named Route for configuring the Router.

NGINX OpenShift Router

NGINX Router is built on top of OpenShift Template Router. Below are the key features of NGINX Router:

• Full support of the Route specification. NGINX Router fully supports the features defined by the Route resource.
• Customization options. Various customization options and extra features are available through environment variables and Route annotations – a common approach in other Router implementations.
• Familiar operational experience. NGINX Router is integrated in OpenShift through the Template Router software, the same software that underpins the default Router implementation. As a result, you get the familiar operational experience, which makes it easier to migrate from the default Router implementation.
• NGINX performance and stability. With NGINX Router you get the performance and reliability of NGINX software.
• Latest NGINX features. We are also excited to bring our new features, such as native support for gRPC load balancing, into the OpenShift Router. As new features are made available in NGINX and NGINX Plus, they can be incorporated into the Router’s capabilities.
• Support for TCP/UDP load balancing. NGINX Router brings support for load balancing TCP/UDP applications, including supporting edge TLS termination and re-encryption for TCP, via a TCP/UDP load balancing extension.
• Support for Prometheus. NGINX Router can optionally expose metrics ready to be collected by Prometheus.
• Advanced features of NGINX Plus. When NGINX Router is used with NGINX Plus, you get the additional benefits of NGINX Plus, such as its monitoring API, dashboard, extended number of metrics for Prometheus and more fine-tuned control over load balancing methods.

How To Get Started

• Read the installation guide for NGINX or NGINX Plus installation instructions.
• See how to use NGINX Router for edge load balancing of an example HTTP application in our Cafe Application example.
• See how to use NGINX Router for edge load balancing of a TCP/UDP application in our TCP/UDP example.

Contacts

We’d like to hear your feedback! If you have any questions or suggestions for NGINX Router, please create an issue on GitHub.

nginx-openshift-router's Issues

[Help request] Does this still work for Openshift 4?

I can found this blog which indicated 3.10
https://blog.openshift.com/introducing-nginx-and-nginx-plus-routers-for-openshift/

How about 4.1?

Not able to expose grpc app though route still showing http1

Describe the bug
Hi All,

I am trying to expose a grpc app through route but could not get success. I build a new image with latest version of nginx 1.9.x.
I created a new router on other node with default router.
To reproduce
Steps to reproduce the behavior:

1. Build a new image with latest nginx 1.9.x version.

2. deploy the router.
   oc adm router router-nginx --images=docker-registry.default.svc:5000/openshift/nginx-openshift-router:0.3 --type='' --selector='router=nginx' --force-subdomain='${name}-${namespace}.testingrouter.com

3. Run a sample grpc app, which is working fine when I try to access through service name, from one of node of OCP cluster. As shown below.
   [root@app2dev ~]# docker run namely/grpc-cli ls grpc-pod.test-secret.svc.cluster.local:50051
   grpc.health.v1.Health
   grpc.reflection.v1alpha.ServerReflection
   helloworld.Greeter

4. I add this env variable to dc of nginx router, showing on ouputt of oc describe dc/router-nginx
   ROUTER_USE_HTTP2: true

5. This is my route
   [root@bastiondev ~]# oc describe route grpc-pod
   Name: grpc-pod
   Namespace: test-secret
   Created: 18 hours ago
   Labels:
   Annotations: nginx.router.openshift.io/grpc=true
   openshift.io/host.generated=true
   Requested Host: grpc-pod-test-secret.appseaadev.iamdg.net.ma
   exposed on router router 18 hours ago
   grpc-pod-test-secret.testingrouter.com exposed on router router-nginx 18 hours ago
   Path:
   TLS Termination: passthrough
   Insecure Policy:
   Endpoint Port: grpc-port

Service: grpc-pod
Weight: 100 (100%)
Endpoints: 172.251.13.223:50051
[root@bastiondev ~]#

6. Already enabled TLS passthrough
   [root@bastiondev ~]# oc get route
   NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
   grpc-pod grpc-pod-test-secret.appseaadev.iamdg.net.ma ... 1 more grpc-pod grpc-port passthrough None

Getting below error.
[root@app2dev ~]# docker run --add-host grpc-pod-test-secret.testingrouter.com:10.80.243.13 namely/grpc-cli ls grpc-pod-test-secret.testingrouter.com
Received an error when querying services endpoint.
ServerReflectionInfo rpc failed. Error code: 14, message: failed to connect to all addresses, debug info: {"created":"@1611892660.199138688","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4143,"referenced_errors":[{"created":"@1611892660.199125817","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}
[root@app2dev ~]#

10.80.243.13, this is the IP on which nginx-router running. and I am adding entry for dns in /etc/hosts with --add-host in above command.

Expected behavior
It should return output like below.
[root@app2dev ~]# docker run --add-host grpc-pod-test-secret.testingrouter.com:10.80.243.13 namely/grpc-cli ls grpc-pod-test-secret.testingrouter.com
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
helloworld.Greeter

It is still serving http1, I tried with bloomrpc to access it, return this error.
{
"error": "14 UNAVAILABLE: Trying to connect an http1.x server"
}

Your environment
Dev environment

• NGINX Router version
• Nginx 1.9.x
• Openshift version
• 3.11
• NGINX or NGINX Plus version
  Nginx
  Additional context
  Add any other context about the problem here. Any log files you want to share.

What I am missing, Could you help to fix this.

--
BR

Amit Bondwal

OpenShift 3.11 router error in nginx.conf in image

Describe the bug
Router won't start on OSE 3.11 due to error in nginx.conf in image

To reproduce
Steps to reproduce the behavior:

1. Follow deployment steps
2. inspect logs of router pod

Expected behavior
A clear and concise description of what you expected to happen.

Your environment

• NGINX Router version v3.11.0+948efc6-96
• Openshift version
  oc v3.11.59
  kubernetes v1.11.0+d4cacc0
• NGINX or NGINX Plus version

Additional context
Add any other context about the problem here. Any log files you want to share.

I0208 20:39:24.593652       1 template.go:297] Starting template router (v3.11.0+948efc6-96)
I0208 20:39:24.639036       1 router.go:481] Router reloaded:
Router was successfully started.
I0208 20:39:24.639139       1 router.go:252] Router is including routes in all namespaces
E0208 20:39:26.465930       1 limiter.go:137] error reloading router: exit status 1
nginx: [emerg] invalid parameter "weight=0" in /var/lib/nginx/conf/nginx.conf:1749
Failed to reload NGINX.

Support snippets annotations

Is your feature request related to a problem? Please describe.

Allows the user to customize NGINX configuration in ways that are not possible via the existing annotations/ env variables. This will enable support for advanced use cases/customization.

Describe the solution you'd like
Use cases:

• As a Router admin, I would like to allow the Router users to customize NGINX configuration for their Routes.
• As a Router user, I would like to customize NGINX configuration for my Routes.

Configuration:

• ROUTER_ENABLE_UNSAFE_ANNOTATIONS . Enables support for unsafe annotations in Routes -- annotations that are not validated by the Router and might lead to invalid NGINX configuration. The default is false.
• unsafe.nginx.router.openshift.io/server-snippets. Sets custom snippets in the server context of the generated NGINX config. If multiple routes are created for the same host and all have this annotation present, the annotation of the primary route will be used. The primary route is a route which name is the alphabetically last among all routes. If some routes have TLS termination enabled, the primary route is a route which name is the alphabetically last among all TLS-enabled routes. Note: The Router doesn't validate this annotation, which might lead to invalid NGINX configuration. Check the Router logs to makes sure that the annotation has been successfully applied.
• unsafe.nginx.router.openshift.io/location-snippets. Sets custom snippets in the location context of the generated NGINX config. Note: The Router doesn't validate this annotation, which might lead to invalid NGINX configuration. Check the Router logs to makes sure that the annotation has been successfully applied.

Note: because the snippets annotations might lead to invalid NGINX configuration, it makes sense to mark them as "unsafe" and mandate that the Router admin must explicitly enable them in the Router."

Behavior:
Assuming that the ROUTER_ENABLE_UNSAFE_ANNOTATIONS env is set to true for the Router.

• if unsafe.nginx.router.openshift.io/server-snippets is present in a Route, its value is inserted in the server context of the generated NGINX config. This applies for HTTP, Passthrough and TCP/UDP Routes. If multiple Routes are created for the same host, only the annotations from the primary Route is used.
• If unsafe.nginx.router.openshift.io/location-snippets is present in a Route, its value is inserted in the location context. This is not applied for Passthrough or TCP/UDP Routes. This should work for both regular and gRPC locations.

Describe alternatives you've considered

• Modifying the template and inserting the necessary NGINX configuration.
• Creating custom annotations -- modifying the template and adding new annotations for a particular use case.

Additional context

• Based on the proposal in #18
• The feature is similar to the snippets annotations in the NGINX Ingress Controller -- https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/configmap-and-annotations.md#snippets-and-custom-templates
• Also, the custom annotations feature in the NGINX Ingress Controller -- https://github.com/nginxinc/kubernetes-ingress/blob/master/docs/custom-annotations.md

ROUTER_USE_HTTP2 not documented

The ROUTER_USE_HTTP2 env variable is not mentioned in the documentation https://github.com/nginxinc/nginx-openshift-router/blob/master/docs/configuration.md although it is present in the template

client_max_body_size uses the default 1M value

Is your feature request related to a problem? Please describe.
We use nginx openshift router to expose grpc services and from time to time we got errors like:

<title>413 Request Entity Too Large</title>

413 Request Entity Too Large

---

nginx/1.15.6

Describe the solution you'd like
add client_max_body_size to the golang template. It can get its value from env. var or define it static but bigger than 1M it has by default.
I can submit patch for that but I wonder in which context it is best to put it http or location?
If it is the latter perhaps it is best the value to come from annotation.

Describe alternatives you've considered
modify the golang template and hardcode it in the http context which will work for me I suppose

Change cafe sample app to avoid running as scc privileged (not run as root)

Is your feature request related to a problem? Please describe.
Openshift does not allow containers to use root by default. Our sample application runs as root. Changing the sample app would eliminate a step that doesn't follow the principle of least privilege

Describe the solution you'd like
Modify the cafe app to run as a user and not root.

Describe alternatives you've considered
A different sample application or modification to the existing cafe app

Additional context
I'll try to submit a PR with the changes.

External routes are not working

I've followed the guide: nginx-oss-router-install.md to deploy the nginx router on a fresh OpenShift 3.11 cluster. The image was downloaded and added to my registry. The deployment itself is there and it is working:

$ oc get dc router
NAME      REVISION   DESIRED   CURRENT   TRIGGERED BY
router    1          1         1         config
$ oc get service router
NAME      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                   AGE
router    ClusterIP   172.30.93.111   <none>        80/TCP,443/TCP,1936/TCP   1d

However now when I can't access any external route. I get the connection refused error.
The router logs don't show any errors besides the one line below in repetition:

W0409 09:09:29.260137 1 reflector.go:272] github.com/openshift/origin/pkg/router/controller/factory/factory.go:112: watch of *v1.Route ended with: The resourceVersion for the provided watch is too old.
W0409 09:17:44.312835 1 reflector.go:272] github.com/openshift/origin/pkg/router/controller/factory/factory.go:112: watch of *v1.Route ended with: The resourceVersion for the provided watch is too old.

HTTP_X_FORWARDED_PORT incorrect on Passthrough routes

Describe the bug
The variable HTTP_X_FORWARDED_PORT=10444 pased to backend contaniners when we use passthrough routes when it must be 80 or 443.

To reproduce
Steps to reproduce the behavior:

1. Create route.
2. Show variables in backend

Expected behavior
HTTP_X_FORWARDED_PORT=80 or HTTP_X_FORWARDED_PORT=443

Your environment

• NGINX Router 0.2
• Openshift 3.11
• NGINX 1.1.5.6

Additional context

Proxy pass ssl_name needs to be adjusted for reencrypt use cases.

Describe the bug
custom ssl_name is needed for reencrypt. This is the case for integrated registry of openshift.

Your environment

• NGINX Router version 0.2
• Openshift version 3.11
• NGINX or NGINX Plus version 1.15

Additional context
i created some fixes in my fork, please review them if they are ready for a pull request.

edit
removed port section, as it is disscussed in #19 and fixed in pullrequest #21