kebe7jun / ropee

A scalable prometheus remote storage adapter for splunk.

Ropee -- A prometheus remote storage adapter for splunk

With this remote storage adapter, Prometheus can use Splunk as a long-term store for time-series metrics.

Docker instructions

A docker image for the splunk storage adapter is available on Docker Hub at kebe/ropee.

Start with docker

# You must edit the following command for your env.
$ docker run -d --name ropee -p 9970:9970 \
    -e LISTEN_ADDR=0.0.0.0:9970 \
    -e SPLUNK_METRICS_INDEX=metrics \
    -e SPLUNK_METRICS_SOURCETYPE=DaoCloud_promu_metrics \
    -e SPLUNK_HEC_TOKEN=asddsa1-12312312-3123-2 \
    -e SPLUNK_HEC_URL=https://192.168.1.1:8088 \
    -e SPLUNK_URL=https://192.168.1.1:8089 \
    -e TIMEOUT=60 \
    -e DEBUG=0 \
    kebe/ropee:latest

Command args

Usage of ./ropee:
  -debug
    	Debug mode.
  -listen-addr string
    	Sopee listen addr. (default "127.0.0.1:9970")
  -log-file-path string
    	Log files path. (default "/var/log")
  -splunk-hec-token string
    	Splunk Http event collector token.
  -splunk-hec-url string
    	Splunk Http event collector url. (default "https://127.0.0.1:8088")
  -splunk-metrics-index string
    	Index name. (default "*")
  -splunk-metrics-sourcetype string
    	The prometheus sourcetype name. (default "DaoCloud_promu_metrics")
  -splunk-url string
    	Splunk Manage Url. (default "https://127.0.0.1:8089")
  -timeout int
    	API timeout seconds. (default 60)

Configuring Splunk

HEC(HTTP Event Collector)

Please follow splunk docs.

Add SourceType for prom metrics

props.conf

[DaoCloud_promu_metrics]
DATETIME_CONFIG = CURRENT
TRANSFORMS-prometheus_to_metric = prometheus_metric_name_value, prometheus_metric_dims
NO_BINARY_CHECK = true
description = Prometheus Metrics.
SHOULD_LINEMERGE = false
pulldown_type = 1
category = Metrics

transforms.conf

[prometheus_metric_name_value]
REGEX = ^([^\s{]+)({[^}]+})? ([-+]?[0-9]*\.?[0-9]+([eE][-+]?[0-9]+)?)
FORMAT = metric_name::$1 ::$2 _value::$3
WRITE_META = true

[prometheus_metric_dims]
REGEX = ([a-zA-Z_][a-zA-Z0-9_]*)="([^"]*)"[, ]*
FORMAT = $1::"$2"
REPEAT_MATCH = true
WRITE_META = true

Add a metric index

Please follow splunk docs.

Configuring Prometheus

...
remote_read:
  - url: "http://127.0.0.1:9970/read"
# for remote read, you should set the basic auth which belongs splunk's user.

remote_write:
  - url: "http://127.0.0.1:9970/write"

Building

go mod download
go run main.go

ropee's Issues

ropee deployment on kubernetes

Hi, thank you for your work on ropee!

We have ropee running in a kubernetes cluster, I do not know if you tried to do that before.
kubernetes requires a readiness / liveness probe, which needs to return 200 (or ropee will be killed by k8s)

We tried to use /metrics as probe, but that caused some issues, maybe because prometheus is using the same endpoint to collect metrics. So we implemented a dumb /health that... returns 200.

Have you ever tried ropee on k8s?
Do you expect /metrics to work as liveness probe call?
Do you have ideas about better healthchecks than just returning 200?

I can help with an upstream PR if you are interested.

support sidecar injector

https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent

Feature request: a controller to inject ropee sidecar for a pod.

Disable SSL verification should be optional and not mandatory.

Hi,

I think it should be an flaggable option to disable SSL vertification.

ropee/storage/client.go

Line 53 in 08054c4

TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, // ignore expired SSL certificates

Best regards,
Ruba

Health check endpoint

Hi @kebe7jun , is there a way to implement health check to ropee service?

Thanks in advance,

logging and sending metrics issue

currently i am not seeing any detailed logging. even set DEBUG environment variable to true. i need to figure out if/why the metrics are failing to send to my splunk instance. perhaps it has something to do with the setBasicAuth implementation which i'd like to ask a question about as well if it can accept a value in the form of Authorization: Splunk token. any help would be greatly appreciated