GithubHelp home page GithubHelp logo

gustavocarita / kubernetes-logging Goto Github PK

View Code? Open in Web Editor NEW

This project forked from inovex/kubernetes-logging

0.0 1.0 0.0 12.58 MB

Demonstration of Kubernetes Logging with Fluentd and ElasticStack

License: Apache License 2.0

Dockerfile 23.08% Makefile 16.57% Shell 20.86% Go 39.49%

kubernetes-logging's Introduction

Kubernetes Logging into Elasticsearch

This repository contains a docker images for a fluentd logshipper, kubernetes configs to deploy a basic elasticsearch cluster with kibana frontend, and documentation. These files should show how to setup a fluentd logshipper as kubernetes daemonset and pipe all container logs into an elasticsearch cluster. The logs are enriched with Metadata like pod_name, pod_id, docker_id. If your kubernetes application logs structured JSON log to STDOUT or STDERR the JSON is interpreted and each field of the JSON is a field in the elasticsearch index.

The content of this repository is based on the example given in the kubernetes repository. https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

Changes need for production

The file kubernetes_config/fluentd-daemonset.yaml contains the configuration for the fluentd logshipper. See the comments in this files for changes in production.

The fluentd_image/Makefile builds the docker container. Change the variables according to your docker registry setup.

The file kubernetes_config/fluentd_config/td-agent.conf contains the fluentd config. You might change it to consume also journald logs or further logs of the kubelet.

Demo

The demo relies on minikube. You have to setup minikube and then run:

minikube start
eval $(minikube docker-env)

The above will start a minikube cluster and setup the docker client in your current shell to the minikube cluster docker daemon.

To run this demo you have to create the image of the fluentd shipper in the minikube VM. Run the following commands in the shell where you ran the commands above:

cd fluentd_image
make build
cd ../es-image
make build
cd ../kibana-image
make build

To setup the Elasticsearch Cluster and a kibana frontend run:

kubectl create -f kubernetes_config/es-controller.yaml
kubectl create -f kubernetes_config/kibana-controller.yaml
kubectl create -f kubernetes_config/es-service.yaml
kubectl create -f kubernetes_config/kibana-service.yaml

To start the fluentd daemonset run:

kubectl create configmap fluentd-config --from-file=kubernetes_config/fluentd_config/td-agent.conf --namespace=kube-system
kubectl create -f kubernetes_config/fluentd-daemonset.yaml

After this setup you can check the pods you have deployed. The command is:

kubectl --namespace=kube-system get pods

The output should look like this:

NAME                             READY     STATUS    RESTARTS   AGE
elasticsearch-logging-v1-15qsf   1/1       Running   0          2m
elasticsearch-logging-v1-278v2   1/1       Running   0          2m
fluentd-logging-mcegp            1/1       Running   0          10s
kibana-logging-v1-mmfv4          1/1       Running   0          2m
kube-addon-manager-minikubevm    1/1       Running   0          4m
kubernetes-dashboard-ms0el       1/1       Running   0          4m

To look at your logs visit the kibana dashboard. You can get the URL to the kibana dashboard via:

minikube service -n kube-system --url kibana-logging

kubernetes-logging's People

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.