fossabot / datakit

DataKit is collection agent for Guance Cloud.

Website • Doc

Read this in other languages.

DataKit is an open source, integrated data collection agent, which provides full platform (Linux/Windows/macOS) support and has comprehensive data collection capability, covering various scenarios such as host, container, middleware, tracing, logging and security inspection.

Key Features

• Support collection of metrics, logging and tracing
• Fully support Kubernetes ecology
• Pipeline: Simple structured data extraction
• Supports third-party data import:
  • Telegraf
  • Prometheus
  • Statsd
  • Fluentd
  • Filebeats
  • Function
  • Tracing related(OpenTelemetry/DDTrace/Zipkin/Jaeger/Skywalking)

Changelog

All DataKit changelog refers to here.

Minimal Requirements

OS	Arch	Install Path
Linux Kernel 2.6.23+	amd64/386/arm/arm64	/usr/local/datakit
macOS 10.12+(Why)	amd64	/usr/local/datakit
Windows 7+/Server 2008R2+	amd64/386	64-bit：C:\Program Files\datakit 32-bit：C:\Program Files(32)\datakit

Install DataKit

We can directly obtain the DataKit installation command from guance cloud. Most of the installation commands seems like that:

• Linux & Mac

DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>" bash -c "$(curl -L https://static.guance.com/datakit/install.sh)"

• Windows

$env:DK_DATAWAY="https://openway.guance.com?token=<YOUR-TOKEN>";Set-ExecutionPolicy Bypass -scope Process -Force; Import-Module bitstransfer; start-bitstransfer -source https://static.guance.com/datakit/install.ps1 -destination .install.ps1; powershell .install.ps1;

• Kubernetes DaemonSet

For more documentations about DataKit installation, see here.

Build From Source

DataKit building relies on some external tools/libs, we must install them all before compile the source code.

• We recommend to build source on Ubuntu 20.04+, other linux distribition may failed to install these dependencies. We do not support build DataKit on Windows.
• Please build the project with make, we haven't testing with Golang/VSCode IDEs

Setup Golang

Install and setup Golang(1.18.3+):

export GOPRIVATE=gitlab.jiagouyun.com/*
export GOPROXY=https://goproxy.cn,direct
export GOPATH=~/go            # depends on your local settings
export GOROOT=~/golang-1.18.3 # depends on your local settings
export PATH=$GOROOT/bin:~/go/bin:$PATH

Install other tools

!!! Do not install these dependencies under datakit source code dir.

• make: apt-get install make
• gcc: apt-get install gcc
• gcc-multilib: apt-get install -y gcc-multilib
• tree: apt-get install tree
• packr2: go install github.com/gobuffalo/packr/v2/[email protected]
• goyacc: go install golang.org/x/tools/cmd/goyacc@latest
• lint related:
  • lint: go install github.com/golangci/golangci-lint/cmd/[email protected]
• eBPF related:
  • clang 10.0+: apt-get install clang
  • llvm 10.0+: apt-get install llvm
  • apt install go-bindata
  • kernel headers
    • apt: apt-get install -y linux-headers-$(uname -r)
• Documentation exporting:
  • waque 1.13.1+

Build

1. Clone code

$ mkdir -p $GOPATH/src/gitlab.jiagouyun.com/cloudcare-tools
$ cd $GOPATH/src/gitlab.jiagouyun.com/cloudcare-tools

$ git clone https://github.com/GuanceCloud/datakit.git   # may be blocked by GFW
$ git clone https://jihulab.com/guance-cloud/datakit.git # jihulab mirror

$ cd datakit

2. Building

make

If building ok, all binaries are generated under dist:

dist
├── [4.0K]  datakit-linux-amd64
│   ├── [ 72M]  datakit
│   └── [4.0K]  externals
│       ├── [ 14M]  logfwd
│       └── [10.0M]  oracle
├── [4.0K]  local
│   ├── [ 26M]  installer-linux-amd64
│   └── [ 228]  version
└── [4.0K]  standalone
    └── [4.0K]  datakit-ebpf-linux-amd64
		        └── [ 38M]  datakit-ebpf

We can build all platforms(Linux/Mac/Windows) with following command:

make testing

Basic Usage

We can use help command to see more usage of DataKit:

datakit help

# Or

./dist/datakit-linux-amd64/datakit help

Contributing

Before contributing, check out some guideline of DataKit:

• Read architecure introduciton
• Read development guideline

Full Documentation

For full documents of DataKit, see

• DataKit Doc

License