nginxinc / nginmesh

Istio compatible service mesh using NGINX

Important Project Notice

This project is no longer under active development. It will be preserved here for the foreseeable future for reference. Please note that the last version released works with Istio 0.7.

NGINX Architecture with Istio Service Mesh

This repository provides an implementation of a NGINX based service mesh (nginMesh). nginMesh is compatible with Istio. It leverages NGINX as a sidecar proxy.

What is Service Mesh and Istio?

Please check https://istio.io for a detailed explanation of the service mesh.

Production Status

The current version of nginMesh is designed to work with Istio release 0.7.1. It should not be used in production environments.

Demo

Recorded demo of nginMesh depoyment.

Architecture

The diagram below depicts how an NGINX sidecar proxy is implemented. The sidecar uses the open source version of NGINX compiled with modules for tracing and monitoring.

The diagram below is an alternative architectural view -

To learn more about the sidecar implementation, see this document.

Quick Start

Below are instructions to quickly install and configure nginMesh. Currently, only Kubernetes environment is supported.

Prerequisites

Make sure you have a cluster with Kubernetes 1.9 or newer. Please see Prerequisites for setting up a kubernetes cluster.

Install Istio and nginMesh

nginMesh requires installation of Istio first.

1. Download and install Istio 0.7.1:

   curl -L https://git.io/getLatestIstio | ISTIO_VERSION=0.7.1 sh -
   

2. Download nginMesh release 0.7.1:

   curl -L https://github.com/nginxinc/nginmesh/releases/download/v0.7.1/nginmesh-0.7.1.tar.gz | tar zx
   

3. Deploy Istio:

   kubectl create -f istio-0.7.1/install/kubernetes/istio.yaml
   

4. Ensure the following Kubernetes services are deployed: istio-pilot, istio-mixer, istio-ingress:

   kubectl get svc  -n istio-system  
   

   istio-ingress            LoadBalancer   10.47.252.40    35.237.173.47   80:32171/TCP,443:32198/TCP                   19h
   istio-mixer              ClusterIP      10.47.251.225   <none>          9091/TCP,15004/TCP,9093/TCP,9094/TCP,9102/TCP,9125/UDP,42422/TCP    19h
   istio-pilot              ClusterIP      10.47.254.118   <none>          15003/TCP,15005/TCP,15007/TCP,15010/TCP,8080/TCP,9093/TCP,443/TCP   19h
   istio-sidecar-injector   ClusterIP      10.47.242.139   <none>          443/TCP                                       9h
   

5. Ensure the following Kubernetes pods are up and running: istio-pilot-* , istio-mixer-* , istio-ingress-* and istio-initializer-*:

   kubectl get pods -n istio-system    
   

   istio-ca-86f55cc46f-nprhw                1/1       Running   0          19h
   istio-ingress-5bb556fcbf-c7tgt           1/1       Running   0          19h
   istio-mixer-86f5df6997-fvzjx             3/3       Running   0          19h
   istio-pilot-67d6ddbdf6-xhztz             2/2       Running   0          19h
   istio-sidecar-injector-5b8c78fd6-8dvq6   1/1       Running   0          9h
   

6. Enable automatic sidecar injection:

   nginmesh-0.7.1/install/kubernetes/install-sidecar.sh
   

7. Verify that the istio-injection label is not applied to the default namespace:

   kubectl get namespace -L istio-injection
   

   NAME           STATUS        AGE       ISTIO-INJECTION
   default        Active        1h        
   istio-system   Active        1h        
   kube-public    Active        1h        
   kube-system    Active        1h
   

Deploy a Sample Application

In this section we deploy the Bookinfo application, which is taken from the Istio samples. Please see Bookinfo for more details.

1. Label the default namespace with istio-injection=enabled:

   kubectl label namespace default istio-injection=enabled
   

2. Deploy the application:

   kubectl apply -f  istio-0.7.1/samples/bookinfo/kube/bookinfo.yaml
   

3. Confirm that all application services are deployed: productpage, details, reviews, ratings:

   kubectl get services
   

   NAME                       CLUSTER-IP   EXTERNAL-IP   PORT(S)              AGE
   details                    10.0.0.31    <none>        9080/TCP             6m
   kubernetes                 10.0.0.1     <none>        443/TCP              7d
   productpage                10.0.0.120   <none>        9080/TCP             6m
   ratings                    10.0.0.15    <none>        9080/TCP             6m
   reviews                    10.0.0.170   <none>        9080/TCP             6m
   

4. Confirm that all application pods are running --details-v1-* , productpage-v1-* , ratings-v1-* , reviews-v1-* , reviews-v2-* and reviews-v3-*:

   kubectl get pods
   

   NAME                                        READY     STATUS    RESTARTS   AGE
   details-v1-1520924117-48z17                 2/2       Running   0          6m
   productpage-v1-560495357-jk1lz              2/2       Running   0          6m
   ratings-v1-734492171-rnr5l                  2/2       Running   0          6m
   reviews-v1-874083890-f0qf0                  2/2       Running   0          6m
   reviews-v2-1343845940-b34q5                 2/2       Running   0          6m
   reviews-v3-1813607990-8ch52                 2/2       Running   0          6m
   

5. Get the public IP of the Istio Ingress controller. If the cluster is running in an environment that supports external load balancers:

   kubectl get svc -n istio-system | grep -E 'EXTERNAL-IP|istio-ingress'
   

6. Open the Bookinfo application in a browser using the following link:

   http://<Public-IP-of-the-Ingress-Controller>/productpage
   

Uninstall the Application

1. To uninstall application, run:

   ./istio-0.7.1/samples/bookinfo/kube/cleanup.sh
   

Uninstall Istio

1. To uninstall the Istio core components:

   kubectl delete -f istio-0.7.1/install/kubernetes/istio.yaml
   

2. To uninstall the initializer, run:

   nginmesh-0.7.1/install/kubernetes/delete-sidecar.sh
   

Limitations

nginMesh has the following limitations:

• TCP and gRPC traffic is not supported.
• Quota Check is not supported.
• Only Kubernetes is supported.

All sidecar-related limitations and supported traffic management rules are described here.

nginmesh's Issues

Where can I find the source codes of nginmesh sidecar?

Hi,

I'm very interested in this project. After going through this repository, I can't find source codes of "nginmesh/istio-nginx-sidecar:0.16-beta", I guess it should be open source as well? Could you please give me a pointer?

Thanks,
Huabing

HTTP Rewrite does not work in release 0.4.0

To reproduce problem:

1. Install istio 0.4.0 in GCP
2. Install Nginmesh 0.4.0 initializer
3. Deploy Bookinfo app
4. Add rewrite rule:
   istioctl create -f ../nginmesh-0.4.0/samples/kubernetes/addons/route-rule-http-rewrite.yaml
5. Check mixer logs for "bookRatings", currently cannot find in logs :
   kubectl logs $(kubectl get pod -n istio-system | grep istio-mixer | awk '{ print $1 }')
   -n istio-system mixer

HTTP Abort Routing Rule does not work as expected

To Reproduce, run in GCP Kubernetes cluster version 1.8.4 with alpha feature enabled:

kubectl apply -f istio-0.4.0/install/kubernetes/istio.yaml
kubectl apply -f nginmesh-0.4.0/install/kubernetes/istio-initializer.yaml
kubectl apply -f istio-0.4.0/samples/bookinfo/kube/bookinfo.yaml
istioctl create -f nginmesh-0.4.0/samples/kubernetes/addons/route-rule-http-abort.yaml
Make few requests to sample application: http://GATEWAY_URL/productpage

Expected: According to rule, based on percent defined under httpFault section in routing configuration file, client requests should get "400 Bad Request" response code for Reviews V3 service proportionally.

Result: Client gets "503 Service Unavailable" response code after rule applied.

HTTP Abort Routing Rule documentation:
https://istio.io/docs/reference/config/traffic-rules/routing-rules.html#abort

HTTP Abort Routing Rule file:
route-rule-http-abort.yaml

install-sidecar.sh script executes with errors

The install-sidecar.sh spits out the following log on my Ubuntu 16.04 machine.

+ echo '# GENERATED FILE. Use with Istio 0.6'
+ cat /root/nginmesh-0.7.2/install/kubernetes/templates/nginmesh-sidecar-injector-configmap.yaml.tmpl
+ sed -i .bak 's|{NGINMESH_IMAGE_HUB}|docker.io|' /root/nginmesh-0.7.2/install/kubernetes/generated/nginmesh-sidecar-injector-configmap.yaml
sed: -e expression #1, char 1: unknown command: `.'
+ sed -i .bak 's|{KAFKA_SERVER}|my-kafka-kafka.kafka:9092|' /root/nginmesh-0.7.2/install/kubernetes/generated/nginmesh-sidecar-injector-configmap.yaml
sed: -e expression #1, char 1: unknown command: `.'
+ sed -i .bak 's|{KAFKA_TOPIC}|nginmesh|' /root/nginmesh-0.7.2/install/kubernetes/generated/nginmesh-sidecar-injector-configmap.yaml
sed: -e expression #1, char 1: unknown command: `.'
+ sed -i .bak 's|{NGX_LOG_LEVEL}|warn|' /root/nginmesh-0.7.2/install/kubernetes/generated/nginmesh-sidecar-injector-configmap.yaml
sed: -e expression #1, char 1: unknown command: `.'
+ sed -i .bak 's|{NGINMESH_VERSION}|0.7.1|' /root/nginmesh-0.7.2/install/kubernetes/generated/nginmesh-sidecar-injector-configmap.yaml
sed: -e expression #1, char 1: unknown command: `.'
+ sed -i .bak 's|{ISTIO_PROXY_INIT}|docker.io/istio/proxy_init:0.7.1|' /root/nginmesh-0.7.2/install/kubernetes/generated/nginmesh-sidecar-injector-configmap.yaml
sed: -e expression #1, char 1: unknown command: `.'

The root cause was found to be the space between the -i and .bak parameters in the sed statements in the generate-sidecar-config.sh script. The sed version on my machine is sed (GNU sed) 4.2.2.

The fix is to change the sed -i .bak ... to sed -i.bak .... I can create a PR for that if you'd like.

Thanks,
Venil

set up istio install directory

(1) move existing installation file under istio
(2) copy istio release template and configure for nginmesh

Support gRPC

Are there any plan for nginmesh to support gRPC?

Thanks,

ServiceGraph does not show sample application graph

In Istio Release 0.4.0 and Nginmesh 0.4.0 Servicegraph does not show sample application graph data.

To Reproduce, run in GCP Kubernetes cluster version 1.8.4 with alpha feature enabled:

kubectl apply -f istio-0.4.0/install/kubernetes/istio.yaml
kubectl apply -f nginmesh-0.4.0/install/kubernetes/istio-initializer.yaml
kubectl apply -f istio-0.4.0/install/kubernetes/addons/prometheus.yaml
kubectl apply -f istio-0.4.0/install/kubernetes/addons/servicegraph.yaml
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o
jsonpath='{.items[0].metadata.name}') 9090:9090 &
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=servicegraph -o jsonpath='{.items[0].metadata.name}') 8088:8088 &
kubectl apply -f istio-0.4.0/samples/bookinfo/kube/bookinfo.yaml
Make few requests to sample application: http://GATEWAY_URL/productpage
Open ServiceGraph in browser : http://localhost:8088/dotviz
Result: No any application flow graph shown.

Mixer_logs.txt
prometheus_logs.txt
servicegraph_log.txt

Add e2e test suite for Kafka Collector

Following should be tested

• request header
• response time
• source and destination uid

add back user agent info in the collector

Istio Auth does not work

To Reproduce, run in GCP Kubernetes cluster version 1.9.4 with alpha feature disabled:

kubectl create -f istio-0.6.0/install/kubernetes/istio-auth.yaml
nginmesh-0.6.0/install/kubernetes/install-sidecar.sh
helm init
nginmesh-0.6.0/install/kafka/install.sh
nginmesh-0.6.0/tools/kafka-add-topics.sh nginmesh
kubectl label namespace default istio-injection=enabled
kubectl apply -f nginmesh-0.6.0/samples/bookinfo/kube/bookinfo.yaml

Make few requests to sample application: http://GATEWAY_URL/productpage

Expected: Page should be successfully loaded.

Result: Page cannot be loaded.

Logs and pods status:

kubectl get pods
NAME READY STATUS RESTARTS AGE
details-v1-64b86cd49-frml5 1/2 CrashLoopBackOff 29 2h
productpage-v1-84f77f8747-8xjsw 1/2 CrashLoopBackOff 29 2h
ratings-v1-5f46655b57-2wqpl 1/2 CrashLoopBackOff 29 2h
reviews-v1-ff6bdb95b-rr2zd 1/2 CrashLoopBackOff 29 2h
reviews-v2-5799558d68-6grl8 1/2 CrashLoopBackOff 29 2h
reviews-v3-58ff7d665b-wqcx4 1/2 CrashLoopBackOff 29 2h

kubectl logs -f productpage-v1-84f77f8747-8xjsw istio-proxy
I0409 21:11:51.541394 1 main.go:111] Starting the agent on kubernetes://productpage-v1-84f77f8747-8xjsw.default at 10.44.1.24
I0409 21:11:51.541590 1 main.go:124] collector address: %v, topic: %vmy-kafka-kafka.kafka:9092nginmesh
I0409 21:11:56.542360 1 client.go:43] listener url: http://istio-pilot.istio-system:15003/v1/listeners/productpage/sidecar~10.44.1.24~productpage-v1-84f77f8747-8xjsw.default~default.svc.cluster.local
F0409 21:11:56.548135 1 client.go:219] Error getting listeners: couldn't get listeners: Get http://istio-pilot.istio-system:15003/v1/listeners/productpage/sidecar~10.44.1.24~productpage-v1-84f77f8747-8xjsw.default~default.svc.cluster.local: read tcp 10.44.1.24:32790->10.47.244.235:15003: read: connection reset by peer

Use new Rust module SDK

Migrate collector module to use Rust SDK

run Istio e2e test

Basic E2E test suites

Istio Auth

This project sounds really interesting. Is there any plan to support Istio Auth or some other method of having TLS between pods managed by the side cars?

Grafana does not show sample application metrics

In Istio Release 0.4.0 and Nginmesh 0.4.0 Grafana does not show sample application metrics.

To Reproduce, run in GCP Kubernetes cluster version 1.8.4 with alpha feature enabled:

1. kubectl apply -f istio-0.4.0/install/kubernetes/istio.yaml
2. kubectl apply -f nginmesh-0.4.0/install/kubernetes/istio-initializer.yaml
3. kubectl apply -f istio-0.4.0/install/kubernetes/addons/prometheus.yaml
4. kubectl apply -f istio-0.4.0/install/kubernetes/addons/grafana.yaml
5. kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o
   jsonpath='{.items[0].metadata.name}') 9090:9090 &
6. kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=grafana -o
   jsonpath='{.items[0].metadata.name}') 3000:3000 &
7. kubectl apply -f istio-0.4.0/samples/bookinfo/kube/bookinfo.yaml
8. Make few requests to sample application: http://GATEWAY_URL/productpage
9. Open Grafana in browser : http://localhost:3000/dashboard/db/istio-dashboard
   Result: No any application data in dashboard.

Mixer_logs.txt
Prometheus_logs.txt
Grafana_logs.txt

update sidecar for Istio 0.7.1

Upgrade Sidecar and implement Helm based template

update sidecar initializer for 0.3.0