Comments (14)
Here's one blog post that suggests a solution, but there may be more.
https://blog.openshift.com/thanos-long-term-storage-for-your-prometheus-metrics-on-openshift/
Also, looking at the Prometheus Storage docs, there's nothing that explicitly says that it should only be used for short-term storage, but it does point out that promehteus itself doesn't do much in the way of operationalizing its storage, and references some thrid-party storage integrations.
https://prometheus.io/docs/prometheus/latest/storage/
from pelorus.
I will start taking a look at this tomorrow. Thanks.
from pelorus.
Theres already been some thanos work in the CoP:
from pelorus.
I'll have a look, thanks!
from pelorus.
I have a working solution on this using nooba for Openshift 4. However, when I try this on the CI CASL cluster, nooba is having issues installing with PV provisioning. Using version 2.0.9 of noobaa, I am able to create a backing store, but I am not able to get noobaa to finish its installation, it gets stuck waiting for something. Looking at the logs, its looking for AWS and Ceph provisioners.
We should be able to use the rest of the solution with any s3 compatible bucket provider. It might be time to start looking at Ceph or to just suggest straight up AWS s3.
from pelorus.
Nooba aggregates S3 API providers, Ceph API and S3 are the default backing types, this cannot be block so if CI CASL doesn't support S3 API compatible storage you'll need something other than Nooba. Thanos is a good fit but the storage needs to be worked out.
from pelorus.
@bbeaudoin ok, I misunderstood what Noobaa was then. @ramius345 I think the default option for this project should be to use MinIO instead.
from pelorus.
Functional code ready (iteration 1), code needs to be merged. Documentation to be completed.
from pelorus.
@tolarewaju3 now that @ramius345 work has been merged, here's what I think it left TODO:
- Come up with an opinionated installation method for MinIO that we can document, that at a minimum includes a TLS config
- Refactor the helm chart to allow the user to configure any Object Store provider for thanos
- Update documentation
from pelorus.
Oh, and one more:
- Repoint grafana to the thanos instance instead of directly to prometheus
from pelorus.
Going to write documentation recommending minio helm chart. The current operator install doesn't work out of the box because of these two issues:
We can update the guide in the future when those are in master
from pelorus.
Test PR is up. Still needs work on documentation & this task:
- Refactor the helm chart to allow the user to configure any Object Store provider for thanos
from pelorus.
Here are my testing steps:
- Deployed the stack per the directions into a single namespace
pelorus
. - Deployed the exporters and loaded some data into Prometheus.
- Verified the data appeared in the dashboard and via PromQL query
- Delete the 2 Prometheus and checked for data. Neither the dashboard or a PromQL query returned any results.
- Created a route for the MinIO instance to verify data was being written to MinIO. Minio shows no data in the buckets.
It appears that no data is being written to MinIO.
from pelorus.
@themoosman Hmm. Ok. I will re-test this when I get some time. When I ran it, it took a bit to populate in minio. But it sounds like you already saw data in pelorus
from pelorus.
Related Issues (20)
- Improper Change Failure Rate metric
- Wrong logic in logs committime for username and token
- Untracked commit is showing wrong number HOT 1
- Install fixed versions of tools with `make dev-env`
- Add option to authenticate against image repository
- Lead Time for Change doesn't show the graph under long interval
- Release of Pelorus Operator version 0.0.8
- mkdocs serve fails to import libcairo on certain mac setups HOT 1
- Simplify command of creating operator test images
- Prow CI should only run against PRs that touch Pelorus code
- sdp:lead_time_for_change:by_commit should include the image sha that it was matched with as a label
- commit_timestamp should include source repo URLs so that users can audit/understand/validate measure calculation
- deploy_timestamp should include image registry URL so that users can verify the source of the deployment information
- requirements.txt files should be automatically generated in CI if the PR is from dependabot HOT 1
- committime exporter not connecting to internal gitea repo HOT 10
- Open a GitHub Issue does not reflect the date in Change Failure Rate dashboard HOT 6
- Grafana doesn't show multiple cluster Pelorus metrics HOT 1
- Lead Time to Change does not reflect on grafana with two type of committime-exporter enabled HOT 1
- Custom Certificate for Jira
- Mean Time to Restore prometheus rules can miss tickets in calculation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pelorus.