guancecloud / datakit

DataKit is collection agent for Guance Cloud

.github

This repository contains the organization profile.

datakit's Issues

image in README.md is dead

installer-linux-arm64-1.1.9-rc7 failed at first try but succeeded at the second try

Host Environment: MacBook 12.0.1 (21A559) with Apple M1
Dockerfile: Attached in the thread
installation logs: Attached below

How to reproduce:

1. replace the token with a valid one in docker file.
2. docker build . -t datakit
3. docker run -it --entrypoint bash datakit
4. ./install.sh
5. echo $? # expect 1
6. ./install.sh
7. echo $? #expect 0

root@dd2e01a6b885:/# ./install.sh
 Detect installer installer-linux-arm64-1.1.9-rc7

* Downloading installer /tmp/dk-installer
##################################################################################################################################################################################################### 100.0%

* Installing DataKit...
tee: /var/log/datakit/install.log: No such file or directory
2021-12-19T04:26:11.306Z	INFO	installer	installer/main.go:245	stoping datakit...
2021-12-19T04:26:11.310Z	WARN	installer	installer/main.go:247	stop service: Failed to stop datakit: exit status 1, ignored
2021-12-19T04:26:11.310Z	DEBUG	installer	installer/main.go:659	deprecated install path /usr/local/cloudcare/dataflux/datakit not exists, ignored
Downloading(datakit)... 17 MB/17 MB
Downloading(   data)... 34 MB/34 MB
2021-12-19T04:26:53.322Z	INFO	installer	installer/main.go:269	Installing version 1.1.9-rc7...
2021-12-19T04:26:53.322Z	WARN	installer	installer/main.go:348	uninstall service: Failed to uninstall datakit: remove /etc/init.d/datakit: no such file or directory, ignored
2021-12-19T04:26:53.323Z	DEBUG	dataway	dataway/dataway.go:208	httpCli: 0x4000244a00
2021-12-19T04:26:53.323Z	INFO	installer	installer/main.go:498	set cloud provider to  ok
2021-12-19T04:26:53.324Z	DEBUG	installer	installer/main.go:501	main config:
default_enabled_inputs = ["cpu", "disk", "diskio", "mem", "swap", "system", "hostobject", "net", "host_processes", "container"]
install_date = 2021-12-19T04:26:53Z
upgrade_date = 0001-01-01T00:00:00Z
namespace = ""
install_version = "1.1.9-rc7"
protect_mode = true
enable_election = false

[[black_lists]]
  hosts = []
  inputs = []

[[white_lists]]
  hosts = []
  inputs = []

[dca]
  enable = false
  listen = "0.0.0.0:9531"
  white_list = []

[http_api]
  rum_origin_ip_header = "X-Forwarded-For"
  listen = "localhost:9529"
  disable_404page = false
  rum_app_id_white_list = []

[io]
  feed_chan_size = 1024
  high_frequency_feed_chan_size = 2048
  max_cache_count = 1024
  cache_dump_threshold = 512
  max_dynamic_cache_count = 1024
  dynamic_cache_dump_threshold = 512
  flush_interval = "10s"
  output_file = ""
  output_file_inputs = []
  enable_cache = false

[dataway]
  urls = ["https://aws-openway.guance.com?token=tkn_I_hope_there_is_no_revision_history"]
  timeout = "5s"
  http_proxy = ""
  max_fail = 20

[logging]
  log = "/var/log/datakit/log"
  gin_log = "/var/log/datakit/gin.log"
  level = "info"
  disable_color = false
  rotate = 32

[global_tags]
  cluster = ""
  project = ""
  site = ""

[environments]
  ENV_HOSTNAME = ""

[cgroup]
  enable = true
  cpu_max = 30.0
  cpu_min = 5.0

[tracer]
  trace_enabled = false
  host = ""
  port = ""
  service = ""
  version = ""
  logs_startup = false
  debug = false
  env = ""

[git_repos]
  pull_interval = "1m"

  [[git_repos.repo]]
    enable = false
    url = ""
    ssh_private_key_path = ""
    ssh_private_key_password = ""
    branch = "master"

2021-12-19T04:26:53.324Z	INFO	config	config/cfg.go:520	here is hostname:
2021-12-19T04:26:53.324Z	INFO	config	config/cfg.go:523	set hostname to dd2e01a6b885
2021-12-19T04:26:53.324Z	INFO	installer	installer/main.go:449	installing service datakit...
2021-12-19T04:26:53.328Z	INFO	installer	installer/main.go:274	starting service datakit...
2021-12-19T04:26:53.355Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/bin/datakit...
2021-12-19T04:26:53.355Z	WARN	config	config/cfg.go:841	remove /usr/local/bin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.355Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/sbin/datakit...
2021-12-19T04:26:53.355Z	WARN	config	config/cfg.go:841	remove /usr/local/sbin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.355Z	DEBUG	config	config/cfg.go:839	remove link /sbin/datakit...
2021-12-19T04:26:53.355Z	WARN	config	config/cfg.go:841	remove /sbin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.356Z	DEBUG	config	config/cfg.go:839	remove link /usr/sbin/datakit...
2021-12-19T04:26:53.356Z	DEBUG	config	config/cfg.go:839	remove link /usr/bin/datakit...
2021-12-19T04:26:53.356Z	WARN	config	config/cfg.go:841	remove /usr/bin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.356Z	INFO	installer	installer/main.go:289	:) Install Success!

	Visit http://localhost:9529/man/changelog to see DataKit change logs.
	Visit http://localhost:9529/monitor to see DataKit running status.
	Visit http://localhost:9529/man to see DataKit manuals.

root@dd2e01a6b885:/# echo $?
1
root@dd2e01a6b885:/# ./install.sh
 Detect installer installer-linux-arm64-1.1.9-rc7

* Downloading installer /tmp/dk-installer
##################################################################################################################################################################################################### 100.0%

* Installing DataKit...
2021-12-19T04:27:35.719Z	INFO	installer	installer/main.go:245	stoping datakit...
2021-12-19T04:27:35.728Z	DEBUG	installer	installer/main.go:659	deprecated install path /usr/local/cloudcare/dataflux/datakit not exists, ignored
Downloading(datakit)... 17 MB/17 MB
Downloading(   data)... 34 MB/34 MB
2021-12-19T04:28:45.151Z	INFO	installer	installer/main.go:269	Installing version 1.1.9-rc7...
2021-12-19T04:28:45.157Z	DEBUG	dataway	dataway/dataway.go:208	httpCli: 0x4000132140
2021-12-19T04:28:45.158Z	INFO	installer	installer/main.go:498	set cloud provider to  ok
2021-12-19T04:28:45.160Z	DEBUG	installer	installer/main.go:501	main config:
default_enabled_inputs = ["cpu", "disk", "diskio", "mem", "swap", "system", "hostobject", "net", "host_processes", "container"]
install_date = 2021-12-19T04:28:45Z
upgrade_date = 0001-01-01T00:00:00Z
namespace = ""
install_version = "1.1.9-rc7"
protect_mode = true
enable_election = false

[[black_lists]]
  hosts = []
  inputs = []

[[white_lists]]
  hosts = []
  inputs = []

[dca]
  enable = false
  listen = "0.0.0.0:9531"
  white_list = []

[http_api]
  rum_origin_ip_header = "X-Forwarded-For"
  listen = "localhost:9529"
  disable_404page = false
  rum_app_id_white_list = []

[io]
  feed_chan_size = 1024
  high_frequency_feed_chan_size = 2048
  max_cache_count = 1024
  cache_dump_threshold = 512
  max_dynamic_cache_count = 1024
  dynamic_cache_dump_threshold = 512
  flush_interval = "10s"
  output_file = ""
  output_file_inputs = []
  enable_cache = false

[dataway]
  urls = ["https://aws-openway.guance.com?token=tkn_I_hope_there_is_no_revision_history"]
  timeout = "5s"
  http_proxy = ""
  max_fail = 20

[logging]
  log = "/var/log/datakit/log"
  gin_log = "/var/log/datakit/gin.log"
  level = "info"
  disable_color = false
  rotate = 32

[global_tags]
  cluster = ""
  project = ""
  site = ""

[environments]
  ENV_HOSTNAME = ""

[cgroup]
  enable = true
  cpu_max = 30.0
  cpu_min = 5.0

[tracer]
  trace_enabled = false
  host = ""
  port = ""
  service = ""
  version = ""
  logs_startup = false
  debug = false
  env = ""

[git_repos]
  pull_interval = "1m"

  [[git_repos.repo]]
    enable = false
    url = ""
    ssh_private_key_path = ""
    ssh_private_key_password = ""
    branch = "master"

2021-12-19T04:28:45.161Z	INFO	config	config/cfg.go:520	here is hostname:
2021-12-19T04:28:45.161Z	INFO	config	config/cfg.go:523	set hostname to dd2e01a6b885
2021-12-19T04:28:45.163Z	INFO	installer	installer/main.go:449	installing service datakit...
2021-12-19T04:28:45.167Z	INFO	installer	installer/main.go:274	starting service datakit...
2021-12-19T04:28:45.198Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/bin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/sbin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /sbin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /usr/sbin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /usr/bin/datakit...
2021-12-19T04:28:45.199Z	INFO	installer	installer/main.go:289	:) Install Success!

	Visit http://localhost:9529/man/changelog to see DataKit change logs.
	Visit http://localhost:9529/monitor to see DataKit running status.
	Visit http://localhost:9529/man to see DataKit manuals.

root@dd2e01a6b885:/# echo $?
0
root@dd2e01a6b885:/#

无法编译

按照README中的操作无法编译，错误如下：

jaeger http协议采集配置

配置文件上说是支持http协议，但没有看到具体的配置，在哪里可以找到配置呢。我的诉求是：otel将链路数据exporter到jaeger（http协议），能否通过这样子exporter到datakit。

Ubuntu22.04 make失败

Linux samsung 5.19.0-41-generic #42~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Apr 18 17:40:00 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Windows11平台安装失败

token invalid format

host_processes.conf文件中的白名单配置未生效

在host_processes.conf文件中添加了进程白名单，但是没生效，还是会把所有进程数据都采集上来。（Linux服务器试过也是有这个问题）

dkebpf.Asset()函数找不到

请问这个dkebpf.Asset()函数在哪里呀，在c目录里面并没有找到对应的方法

can't find git path in main branch in datakit

is a bug?

pull main branch source code , can't find datakit/internal/git lib path

kafka handle

kafkamq 提供一种插件机制：将数据（[]byte）通过 HTTP 发送到外部handle，经过处理后再通过response返回行协议的json格式数据。实现定制化数据。

增加如下配置：（以最终配置为准）

• http url string
• message_points int 一次发送的消息点数
• debug bool 值, 当开启debug功能， message_points 则无效，如果开启debug模式，则将原始byte数据发送，不再进行消息合并。
• threads int 多线程工作
• is_response_point 是否将行协议数据发送回来
• pipeline 脚本
• header_check 特殊的头部检测（bfy定制化，并非通用）
• 等等

外部插件有一些约束：

• kafkamq 接收数据但不负责解析
• 外部插件解析后的数据可以通过 dk api 发送，也可以返回到 kafkamq 再发送到观测云
• 通过 response 返回到 kafkamq 必须是 行协议格式。
• 外部插件收到数据，无论解析失败与否 都应该返回200。
• kafkamq发送数据到外部插件如果出现timeout，端口不存在等。会尝试重连。不再消费kafka中的消息。

root用户名更改后无法安装安全巡检

linux服务器因等保要求，将root用户名改了，导致安全巡检无法安装。

prometheus数据接入失败

Hello，按照这个文档想接入更多的prometheus数据 https://docs.guance.com/integrations/prom/#prometheus-exporter
配置了两个外部地址

monitor上也看到了响应的input

但是观测云指标集里一直看不到数据上来

用了 datakit debug --prom-conf prom.conf 能看到有input，datakit容器没有任何的错误日志，datakit_filter_point_dropped_total 似乎默认也没有被采集无法查询到

为了防止1000个指标过多，relabel到17个之后还是不行

请问还有别的排障手段吗，datakit跟gateway的网络没问题，其他指标一直都正常写入