TNS Observability Demo
A simple three-tier demo application, fully instrumented with Prometheus, Jaeger and Loki logging.
The "TNS" name comes from "The New Stack", where the original demo code was used for an article.
Prerequisites
There are a set of tools you will need to download and install.
Docker
This demo assumes you have Docker installed. Follow instructions here for more details.
k3d
To run this demo, you need a Kubernetes cluster. While the demo should work against any
Kubernetes cluster, these docs will assume a locally running k3d
cluster. Download and
install k3d
from here.
kubectl
kubectl
is used to interact with Kubernetes clusters. Follow the instructions
here to install it.
tanka
Tanka uses the Jsonnet language to interact with Kubernetes, via the kubectl
tool.
Download and install it from here.
jsonnet-bundler
Jsonnet bundler downloads Jsonnet dependencies. Download and install it from here.). Rename the downloaded binary to jb and move it to the location where $PATH points. Also make sure the binary is executable:
$ chmod +x /usr/local/bin/jb
Instructions
If you wish to use a Kubernetes cluster other than a local k3d
one, please adjust these
instructions accordingly.
- Clone TNS repository
$ git clone https://github.com/grafana/tns
$ cd tns
- Install K3D Cluster
$ ./create-k3d-cluster
$ export KUBECONFIG=$(k3d kubeconfig write tns)
If you see any error like,
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock
then try to add your user to docker
group to run docker without sudo
.
$ sudo usermod -aG docker <your-user>
you should logout and login again for the changes to take effect.
- Install TNS applications
This step will ask you to confirm
yes
four times.
$ ./install
- Wait It will take some time to install the demo - there's a lot of downloading to do. It is not unreasonable for it to take over ten minutes for everything to download then start up.
This command will show you the status of the cluster:
$ kubectl get pods -A
All pods should be listed as either running
or completed
. If this is the case,
your cluster should be ready for use.
Accessing the Demo
You should now be able to access the demo via http://localhost:8080/.
Demoable things
Metrics -> Logs -> Traces
- Go to the TNS dashboard
- Zoom in on a section with failed requests if you are so inclined
- Panel Drop Down -> Explore
- Datasource Drop Down -> Loki
- Choose a log line with a traceID -> Tempo
Metrics -> Traces -> Logs
- Go to Explore
- Choose Datasource prometheus-exemplars
- Run this query
histogram_quantile(.99, sum(rate(tns_request_duration_seconds_bucket{}[1m])) by (le))
- Click an exemplar
- Click the log icon on a span line
LogQLV2
- Go to Explore
- Choose Datasource Loki
- Run this query
{job="tns/app", level="info"} | logfmt | status>=500 and status <=599 and duration > 50ms
- Choose a log line with a traceID -> Tempo
Reviewing the Tanka Code
This installation will have created a tanka
directory in your TNS checkout. This
directory contains all of the Jsonnet resources used to install this demo.
You will now have a tanka
directory within your checkout that contains all of the
Jsonnet resources that were needed to deploy this monitoring stack. To find out more
about Tanka, visit https://tanka.dev.
Disabling/enabling your cluster
Should you wish to disable your cluster, use this command:
$ k3d cluster stop tns
To re-enable it, do this:
$ k3d cluster start tns
Removing the Cluster
Once you have finished with the cluster, this should remove it and leave you ready to recreate it on another occasion:
$ k3d cluster delete tns
$ rm -rf tanka