Comments (5)
I would guess that the querier component is not correctly seeing the blocks. Perhaps there is a polling issue? Can you check metrics such as tempodb_blocklist_length
to see if the querier is aware of blocks in the backend?
from tempo.
Following are metric values regarding blocklist:
# HELP tempodb_blocklist_poll_duration_seconds Records the amount of time to poll and update the blocklist.
# TYPE tempodb_blocklist_poll_duration_seconds histogram
tempodb_blocklist_poll_duration_seconds_bucket{le="0"} 0
tempodb_blocklist_poll_duration_seconds_bucket{le="60"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="120"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="180"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="240"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="300"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="360"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="420"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="480"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="540"} 214
tempodb_blocklist_poll_duration_seconds_bucket{le="+Inf"} 214
tempodb_blocklist_poll_duration_seconds_sum 13.568723699999998
tempodb_blocklist_poll_duration_seconds_count 214
# HELP tempodb_blocklist_tenant_index_age_seconds Age in seconds of the last pulled tenant index.
# TYPE tempodb_blocklist_tenant_index_age_seconds gauge
tempodb_blocklist_tenant_index_age_seconds{tenant="single-tenant"} 0
# HELP tempodb_blocklist_tenant_index_builder A value of 1 indicates this instance of tempodb is building the tenant index.
# TYPE tempodb_blocklist_tenant_index_builder gauge
tempodb_blocklist_tenant_index_builder{tenant="single-tenant"} 1
# HELP tempodb_blocklist_tenant_index_errors_total Total number of times an error occurred while retrieving or building the tenant index.
# TYPE tempodb_blocklist_tenant_index_errors_total counter
tempodb_blocklist_tenant_index_errors_total{tenant="single-tenant"} 1
from tempo.
Since we are not seeing a tempodb_blocklist_length
that means the querier believes there are no blocks in the backend and that is why you are not seeing older traces.
Can you check the configured paths to see if they have any blocks? or review the logs to see if you catch any polling errors?
I'm not familiar with Windows so it's difficult to comment on the configured paths. Should they be /
s?
from tempo.
These are error and warn logs:
level=error ts=2024-04-11T07:55:32.8295609Z caller=poller.go:225 msg="failed to pull bucket index for tenant. falling back to polling" tenant=single-tenant err="does not exist"
level=warn ts=2024-04-11T07:55:32.8928056Z caller=modules.go:259 msg="metrics-generator is not configured." err="no metrics_generator.storage.path configured, metrics generator will be disabled"
level=warn ts=2024-04-11T07:55:32.8938052Z caller=modules.go:287 msg="Worker address is empty in single binary mode. Attempting automatic worker configuration. If queries are unresponsive consider configuring the worker explicitly." address=127.0.0.1:9095
level=warn ts=2024-04-11T07:55:32.9104533Z caller=wal.go:116 msg="unowned file entry ignored during wal replay" file=blocks err=null
Checked tempo data folder and it has data. Following is the structure:
├───tempo
│ ├───blocks
│ │ └───single-tenant
│ │ ├───01a0be7d-9da3-45a1-953b-96e07b795305
│ │ ├───01cd7f2d-b1a8-4caa-94f8-64b15bae01bd
......................................................
│ └───wal
│ ├───551f37dd-38ef-4590-bd2a-f9b9a9996380+single-tenant+vParquet3
│ └───blocks
│ └───single-tenant
│ ├───596f4cfe-e3e7-4cca-9e6b-17e30505eeaf
│ ├───7c4a8965-7250-43d8-9062-72b7184c6bd4
......................................................
Tried with \
s in path and the result is the same.
Also another issue is that tempo data folder size is only growing. It looks like old files are not deleted.
from tempo.
Also another issue is that tempo data folder size is only growing. It looks like old files are not deleted.
Right. It seems like all components are failing to discover the blocks for your tenant. This means querying/compaction/retention will not work. Perhaps there is a bug in the local backend on windows.
from tempo.
Related Issues (20)
- Code(499) desc = context canceled HOT 3
- tempo memcached via helm deployment gives i/o timeout HOT 1
- Trace data is being generated even though there are no request from service A to service B HOT 3
- Tempo-cli trace-summary issue HOT 5
- [DOC] Tempo 2.5 release notes
- Can't use https while accessing minio s3. I have deployed k8s on AKS and using minio to store traces on onprem . I am using https:// but can't access s3. HOT 1
- Tempo/Vulture: GetObject S3 API operation - costs. HOT 2
- Tags Endpoint: Incorrect intrinsics returned HOT 5
- User-configurable overrides api Documentation HOT 1
- Need a way to stop tempo from using dual stack urls for s3 backend HOT 13
- metrics-generator stops working HOT 2
- Search query TraceQL is truncated to 1024 characters HOT 1
- S3 Not Reusing Connections HOT 3
- Add querier metrics for requests executed metric to tempo operational dashboard HOT 1
- Tempo (v2.5.0) Distributor missing tempo_receiver_accepted_spans HOT 1
- The number of live traces spikes on one ingester HOT 3
- Streaming gRPC does not avoid 'message larger than max' error HOT 4
- Timeout & cache on metrics generator local-blocks processor HOT 7
- Create a configurable list of TraceQL queries that are immediately 400'ed HOT 3
- Unable to get Service list within time range even if 'start' and 'end' parameters are sent HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tempo.