Splunk allows the collection and analyzes of high volumes of machine-generated data (e.g. application logs). Once the data becomes indexes in Splunk, one can build reporting dashboard and alerts based of specific search. For instance, one can build a dashboard for application crashes, or failures to handle incoming request and track this over time. Splunk provides many integrations that makes it very easy to collect logs from a varied of sources. In this article, we will examine how to collect logs from cloud native applications running on Kubernetes.
1. Splunk Operator
We can easily setup Splunk on Kubernetes using the official operator - link.
First, create a Kubernetes namespace to host the pods of the Splunk operator as well as Splunk itself.
$ kubectl create namespace monit
Second, install Splunk official operator in the newly created namespace as follows
$ kubectl apply -f https://github.com/splunk/splunk-operator/releases/download/1.0.2/splunk-operator-install.yaml -n monit
customresourcedefinition.apiextensions.k8s.io/clustermasters.enterprise.splunk.com created
customresourcedefinition.apiextensions.k8s.io/indexerclusters.enterprise.splunk.com created
customresourcedefinition.apiextensions.k8s.io/licensemasters.enterprise.splunk.com created
customresourcedefinition.apiextensions.k8s.io/searchheadclusters.enterprise.splunk.com created
customresourcedefinition.apiextensions.k8s.io/standalones.enterprise.splunk.com created
serviceaccount/splunk-operator created
role.rbac.authorization.k8s.io/splunk:operator:namespace-manager created
rolebinding.rbac.authorization.k8s.io/splunk:operator:namespace-manager created
deployment.apps/splunk-operator created
After few seconds, the operator will become ready to use, you can check the Pod status with
$ kubectl get pods -n monit
NAME READY STATUS RESTARTS AGE
splunk-operator-75b749554-vcpp8 1/1 Running 0 10s
2. Splunk Standalone
Now, we can deploy Splunk using this operator
$ cat <<EOF | kubectl apply -n monit -f -
apiVersion: enterprise.splunk.com/v2
kind: Standalone
metadata:
name: s1
finalizers:
- enterprise.splunk.com/delete-pvc
EOF
standalone.enterprise.splunk.com/s1 created
After few moments, Splunk Pods will become available and ready to be used. We can check their status as follows:
$ kubectl get pods -n monit
NAME READY STATUS RESTARTS AGE
splunk-default-monitoring-console-0 1/1 Running 0 3m19s
splunk-operator-75b749554-vcpp8 1/1 Running 0 6m38s
splunk-s1-standalone-0 1/1 Running 0 5m56s
3. Splunk credentials
To get the credentials to access Splunk Web UI with kubectl we can print the secret created as part of the deployment of Splunk as follows:
$ kubectl get secret splunk-monit-secret -n monit -o jsonpath='{.data}'
{"hec_token":"Mzc3RjJCRTMtNTE4Ni05MzAyLTY1NTUtRjBBMTk1MDIxQTVE","idxc_secret":"bTI4cnRqSVhpaVI0cGdpUGRFR3lKSjV2","pass4SymmKey":"RjNrYURia3Zyck11ZTlkMEFvYk5Pd
29F","password":"bzMwZnhSRkJYdTlLRjlRUnU5enFPM3ht","shc_secret":"OXpOaTY5MmdrVkt4Nm5xQ3FzR0RlMkQ4"}
Note: the secret name splunk-monit-secret is a composed name of the --secret. If splunk is deployed in the default namespace, the secret name will be splunk-default-secret
To get one specific secret, for instance the Splunk Web UI password of the admin user we can do
$ kubectl get secret splunk-monit-secret -n monit -o jsonpath='{.data.password}' |base64 -d
o30fxRFBXu9KF9QRu9zqO3xm
Then we can access the Web UI by setting up port-forwarding to Splunk as follows
$ kubectl port-forward splunk-s1-standalone-0 8000 -n monit
Forwarding from 127.0.0.1:8000 -> 8000
Forwarding from [::1]:8000 -> 8000
4. Splunk Connect
To be able to send logs to our Splunk deployment we need to get credentials. In our case, we specifically need HEC (HTTP Event Collector) token.
To get the HEC token using kubectl
$ kubectl get secret splunk-monit-secret -n monit -o jsonpath='{.data.hec_token}' |base64 -d
377F2BE3-5186-9302-6555-F0A195021A5D
Configuration file
We need to pupulate a custom version of values.yaml with information specific to our Splunk instance like hostname and HEC token.
We need to create some environment variables to use when filling in the values.yaml file.
-
get Get splunk server address, use DNS name . or just
$ hostname="splunk-s1-standalone-service"
-
get the Splunk HEC token into a variable
$ token=`kubectl get secret splunk-monit-secret -n monit -o jsonpath='{.data.hec_token}' |base64 -d`
-
get the Splunk admin password into a variable
$ password=`kubectl get secret splunk-monit-secret -n monit -o jsonpath='{.data.password}' |base64 -d`
-
choose the index name to be used by Splunk to host the logs
$ index="main"
-
pick a filename where the values will created.
$ file=$(mktemp /tmp/splunk-connect-values.XXXXXX)
-
Create the values file and configure each section.
For instance, the bare minimum file would look like this where we disable sending to Splunk the kubernetes objects and metrics but only allow logging messages to be sent.
$ cat >"${file}" << EOF
global:
splunk:
hec:
host: ${hostname}
port: 8088
token: ${token}
protocol: https
indexName: ${index}
insecureSSL: true
splunk-kubernetes-logging:
enabled: true
containers:
logFormat: '%Y-%m-%dT%H:%M:%S.%N%:z'
logFormatType: cri
logs:
applogs:
from:
pod: '*'
multiline:
firstline: /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}[-+]\d{4}/
separator: ""
splunk-kubernetes-objects:
enabled: false
splunk-kubernetes-metrics:
enabled: false
EOF
Installation with Helm
To be able ot install Splunk Connect with Helm, we to indicate to Helm where to find the charts. For this, add the Splunk Connect github repository to the list of local Help repositories
$ helm repo add splunk https://splunk.github.io/splunk-connect-for-kubernetes/
$ helm repo update
Now we can install Splunk Connect on the monitoring namespace using the custom values file we created in the previous section.
$ helm install splunkconnect -n monit -f "${file}" splunk/splunk-connect-for-kubernetes
NAME: splunkconnect
LAST DEPLOYED: Fri Dec 29 16:22:53 2023
NAMESPACE: monit
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
███████╗██████╗ ██╗ ██╗ ██╗███╗ ██╗██╗ ██╗██╗
██╔════╝██╔══██╗██║ ██║ ██║████╗ ██║██║ ██╔╝╚██╗
███████╗██████╔╝██║ ██║ ██║██╔██╗ ██║█████╔╝ ╚██╗
╚════██║██╔═══╝ ██║ ██║ ██║██║╚██╗██║██╔═██╗ ██╔╝
███████║██║ ███████╗╚██████╔╝██║ ╚████║██║ ██╗██╔╝
╚══════╝╚═╝ ╚══════╝ ╚═════╝ ╚═╝ ╚═══╝╚═╝ ╚═╝╚═╝
Listen to your data.
Splunk Connect for Kubernetes is spinning up in your cluster.
After a few minutes, you should see data being indexed in your Splunk.
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
Warning: Disabling TLS will send the data unencrypted and will be vulnerable to MiTM attacks
After successfully deploying Splunk Connect an index called main will be created, we can check this in the Splunk UI (at http://localhost:8000 with login admin:${password})
Deploy Sample App in test1 namespace:
$ kubectl create ns test1
$ kubectl create deployment python-flask --image=tushardashpute/python_app:v1 -n test1 --port=5000
deployment.apps/python-flask created
$ kubectl port-forward deploy/python-flask -n test1 5000:5000
Forwarding from 127.0.0.1:5000 -> 5000
Forwarding from [::1]:5000 -> 5000
Now access the app using : localhost:5000
This app will log below messages:
app.logger.info('This is an INFO message')
app.logger.debug('This is a DEBUG message')
app.logger.warning('This is a WARNING message')
app.logger.error('This is an ERROR message')
app.logger.critical('This is a CRITICAL message')
return 'Hello, World!'
We can also check logs from our Pods are forwarded properly to splunk