Comments (12)
I've drafted a PR to fix the bug grafana/helm-charts#3236. I've tested it locally both with new and legacy overrides, but more testing is welcome. Sorry for the inconveniences.
from tempo.
@mapno How is one supposed to use overrides currently with helm, when using tempo-distributed? the tempo-distributed helm chart expects the value be
overrides: |
overrides:
foo: bar
And what is in the documentation is:
overrides:
defaults:
foo: bar
doesn't work, as it results in an error
template: tempo-distributed/templates/_helpers.tpl:191:14: executing "tempo.overridesConfig" at <.Values.overrides>: wrong type for value; expected string; got map[string]interface {}
When using the legacy method with the legacy names (e.g. ingestion_rate_strategy
), in my deployment logs there is the following message:
msg="Overrides config type mismatch" err="per-tenant overrides config type does not match static overrides config type" config_type=legacy static_config_type=new
I am also using the latest helm chart version.
from tempo.
Hi! Do you have per-tenant overrides? You need to use the same format in both overrides configs.
from tempo.
Hello!
I have multi-tenancy disabled in the same config file
from tempo.
indeed, I am encountering this issue, too. It is quite confusing. Documentation is no where to be found how tempo tells which configuration is "new" and which is "legacy".
from tempo.
I get this too using the tempo-distributed helm chart and:
global_overrides:
defaults:
metrics_generator:
processors:
- span-metrics
processor:
span_metrics:
histogram_buckets: [null]
dimensions:
- kubernetes_cluster
- kubernetes.cluster
- http.status_code
from tempo.
Documentation is no where to be found how tempo tells which configuration is "new" and which is "legacy".
That's fair. We probably didn't add the best documentation of the change. As a rule of thumb: long, unindented params are legacy
(eg. metrics_generator_forwarder_queue_size
), indented params are new
(eg. metrics_generator.forwarder.queue_size
).
With the new config, a new key defaults
was also added after the overrides
key, to be more explicit of the contents of the config block.
Finally, there is a command in the tempo-cli that allows you to convert legacy config to new overrides config.
from tempo.
Thank you for the feedback on the docs. I've created a doc issue to make sure that we capture these changes and address the issues you've had.
from tempo.
@randrewy @AlexDCraig were you able to solve this somehow?
from tempo.
@markustoivonen I had to use old format despite of what docs are saying. So my overrides section looks like this
overrides:
metrics_generator_processors: [service-graphs, span-metrics]
metrics_generator_processor_local_blocks_complete_block_timeout: "1h"
despite the fact I use latest stable version available atm (2.5.0).
I am not using helm, so this is easy for me to do
from tempo.
@mapno I know we need to update the docs to address new vs legacy. With your fix, will the example in the docs linked in. previous comments still work?
from tempo.
@mapno I know we need to update the docs to address new vs legacy. With your fix, will the example in the docs linked in. previous comments still work?
The examples are now correct, they didn't work before. Yes, legacy overrides still work too :)
from tempo.
Related Issues (20)
- No query results for older telemetry HOT 3
- Tenant error when S3 bucket analytics turned on for S3 backend storage bucket HOT 1
- [DOC] Clarify endpoint access for consistent hash rings doc
- generator push to prometheus pushgateway always fails "HTTP status 400 Bad Request: snappy: corrupt input" HOT 2
- [DOC] Review and update the Parquet docs
- AWS S3 HTTP 4xx client errors HOT 13
- Adjust client/server span times to account for clock skew HOT 3
- Error on kubernetes (microk8s) with persistence enabled HOT 7
- Data retention for 30 days in grafana tempo do not work HOT 5
- User-configurable overrides refuses span-metrics-size as valid processor
- tempo-distributed does not support OpenShift installation out of the box because of missing securitycontextconstraint HOT 2
- aws s3 storage backend:List HOT 2
- pusher failed to consume trace data due to "context canceled" HOT 2
- Monolithic deployment with s3 storage fails with "mkdir /var/tempo/wal: permission denied" HOT 1
- Periodic errors during searching for traces HOT 1
- [DOC] Release notes for 2.6
- failed to get trace with id: a343f1a19e969ea3003b56dfe3802069 Status: 499 status code 499 Body: context canceled HOT 1
- error querying ingesters in Querier.SearchTags: failed to execute f() for [::]:9095: rpc error: code = Unimplemented desc = unknown service tempopb.Querier HOT 4
- Add a limit that caps the maximum allowed size of an attribute
- [tempo-distributed, grafana] How to enable traceQLStreaming getting error "rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/plain; charset=utf-8""
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tempo.