cleafy / promqueen Goto Github PK

prometheus offline data recorder and backfiller

License: Apache License 2.0

Go 89.68% Ruby 3.31% Dockerfile 3.13% Shell 3.88%

prometheus prometheus-metrics prometheus-utility prometheus-monitoring-solution recorder replayer golang go devops devops-tools

promqueen's People

Contributors

Stargazers

Watchers

Forkers

parry84 impimp il-mattone sessa93 etsangsplk jsmette hanwoody silvermind zer0c14 rakeshr-18 sebastianpiskorski shivp0616 thypon acdmail

promqueen's Issues

Mask the password on http basic auth

It should be possible to mask the credentials in the case we use http basic on http requests.

Progress bar for promplay

It would be nice to have a progress bar indicating the progress over the metrics files.
Have a look at:
https://github.com/cheggaaa/pb/tree/v2

Any chance this awesome project will be adapted to work with a 2.x version of prometheus?

Unfortunately I was not able to compile this with what is currently the storage layer for prometheus :-(.

$~/go/src/github.com/Cleafy/promqueen/bin/promplay$ go build
main.go:22:2: cannot find package "github.com/prometheus/prometheus/storage/local" ...

Are there any plans to get this project running again or am I just doing it wrong?

[Question] How to use promplay?

First of all, thank you for this app.

My goal

Export data from Prometheus which runs on server X
Import data to Prometheus which runs on server Y

How to replicate

Server X runs Prometheus on port 9090, server Y on port 9091

Export data with promrec (5s interval, 3 iterations)

promrec --output="./data/metrics" -u service.name=http://localhost:9090/metrics -i 5s -n 3

Output file is metrics

Using vim to find and replace the port numbers from 9090 to 9091 in metrics file
```
:%s/:9090/:9091/g
```
Import to server Y This is where I'm having trouble, assuming I'm in a different directory
```
promplay --nopromcfg --dir=.  --storage.path="data"
```

Output:

INFO[0000] Loading series map and head chunks...         source="storage.go:373"
WARN[0000] Persistence layer appears dirty.              source="persistence.go:815"
WARN[0000] Starting crash recovery. Prometheus is inoperational until complete.  source="crashrecovery.go:40"
WARN[0000] To avoid crash recovery in the future, shut down Prometheus with SIGTERM or a HTTP POST to /-/quit.  source="crashrecovery.go:41"
INFO[0000] Scanning files.                               source="crashrecovery.go:55"
INFO[0000] File scan complete. 0 series found.           source="crashrecovery.go:83"
INFO[0000] Checking for series without series file.      source="crashrecovery.go:85"
INFO[0000] Check for series without series file complete.  source="crashrecovery.go:130"
INFO[0000] Cleaning up archive indexes.                  source="crashrecovery.go:402"
INFO[0000] Clean-up of archive indexes complete.         source="crashrecovery.go:493"
INFO[0000] Rebuilding label indexes.                     source="crashrecovery.go:501"
INFO[0000] Indexing metrics in memory.                   source="crashrecovery.go:502"
INFO[0000] Indexing archived metrics.                    source="crashrecovery.go:510"
INFO[0000] All requests for rebuilding the label indexes queued. (Actual processing may lag behind.)  source="crashrecovery.go:529"
WARN[0000] Crash recovery complete.                      source="crashrecovery.go:152"
INFO[0000] 0 series loaded.                              source="storage.go:378"
Frames processed: [-----------------------------------------------------------------------------------------------] 0s 100.00%
INFO[0000] Stopping local storage...                     source="storage.go:396"
INFO[0000] Stopping maintenance loop...                  source="storage.go:398"
INFO[0000] Maintenance loop stopped.                     source="storage.go:1259"
INFO[0000] Stopping series quarantining...               source="storage.go:402"
INFO[0000] Series quarantining stopped.                  source="storage.go:1701"
INFO[0000] Stopping chunk eviction...                    source="storage.go:406"
INFO[0000] Chunk eviction stopped.                       source="storage.go:1079"
INFO[0000] Checkpointing in-memory metrics and chunks...  source="persistence.go:612"
INFO[0000] Done checkpointing in-memory metrics and chunks in 9.896598ms.  source="persistence.go:639"
INFO[0000] Checkpointing fingerprint mappings...         source="persistence.go:1480"
INFO[0000] Done checkpointing fingerprint mappings in 4.744828ms.  source="persistence.go:1503"
INFO[0000] Local storage stopped.                        source="storage.go:421"

Zero series were loaded, but I can still see the data directory

data
├── VERSION
├── archived_fingerprint_to_metric
├── archived_fingerprint_to_timerange
├── heads.db
├── labelname_to_labelvalues
├── labelpair_to_fingerprints
└── mappings.db

How do I import this data to Prometheus? I thought that the below example should be the expected output, and then I should be able to copy the hashes to the proper Prometheus data folder.

data
├── 01E9WPHANY7DX13M80X3SM909T
├── 01E9ZHMENNSDYHPJRP0KG8G5SQ
├── 01E9ZHMHZJ1CSR8BV2AKD6KS17
├── 01E9ZPNFQRYE43VEZ60Z7ZVB31
├── lock
├── queries.active
├── snapshots
└── wal

May you please share the proper usage of this great tool ?

cannot find package "github.com/prometheus/prometheus/storage/local

hello, i am getting error message when use go get for brand new install. should I revert back to some historical version of prometheus, or i miss sth. (sorry very new to golang)

$ go get github.com/Cleafy/promqueen/bin/promplay
package github.com/prometheus/prometheus/storage/local: cannot find package "github.com/prometheus/prometheus/storage/local"

checking prometheus storage folder i do not see any file as "local*" too.
my platform ubuntu 16.04.3 LTS x86_64, i try both go version 1.6 (ubuntu apt repository), and go1.9.1.

thanks

Generated data layout and usage instructions

Hi there,

I've been playing around with promrec and promfill after finding a reference on (prometheus/prometheus#535) but I'm struggling to get the tools working for me.

I notice that the directory and file layout that results from running promplay looks different than an empty Prometheus data store just created by bringing Prometheus up (I tried version 1.7.1 and 1.8.2).

After running Promplay:

[XXX@XXX data]$ tree
.
├── archived_fingerprint_to_metric
│   ├── 000002.log
│   ├── CURRENT
│   ├── LOCK
│   ├── LOG
│   └── MANIFEST-000003
├── archived_fingerprint_to_timerange
│   ├── 000002.log
│   ├── CURRENT
│   ├── LOCK
│   ├── LOG
│   └── MANIFEST-000003
├── DIRTY
├── heads.db
├── labelname_to_labelvalues
│   ├── 000001.log
│   ├── CURRENT
│   ├── LOCK
│   ├── LOG
│   └── MANIFEST-000000
├── labelpair_to_fingerprints
│   ├── 000001.log
│   ├── CURRENT
│   ├── LOCK
│   ├── LOG
│   └── MANIFEST-000000
└── VERSION

Fresh Prometheus installation

/prometheus/data # ls *

99:
6bf7c4f0744d30.db  81c763c5f4d943.db  8d67781712ca9b.db  a0d965e6535daf.db  b394c82595132c.db  eb84fea15ae0a8.db  f4b4c142823c09.db

9a:
54201d6ae73552.db  6806be24a5ce00.db

de:
0e7d051f319666.db  f43a6192e1be9c.db

df:
02d74982a0fffa.db  10cbca521ed1cd.db  17a83d5b652ca7.db  5a507fbaf0dfc3.db  a23eb2338abee9.db  aaa3bb99297f27.db  aaf0e130a90c18.db  e4f026f3b83be1.db  f8b3d8510ca802.db

It would be much appretiated if you could show an example of filling the database with random metrics (or give a description how to) and then show where to map the resulting data stores on an existing Prometheus installation.

cleafy / promqueen Goto Github PK

promqueen's People

Contributors

Stargazers

Watchers

Forkers

promqueen's Issues

Mask the password on http basic auth

Progress bar for promplay

Any chance this awesome project will be adapted to work with a 2.x version of prometheus?

[Question] How to use promplay?

cannot find package "github.com/prometheus/prometheus/storage/local

Generated data layout and usage instructions

Correct typo in --mexMemoryChunkToPersist

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

Jobs