grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.

Grafana Tempo is an open source, easy-to-use and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki.

Tempo is Jaeger, Zipkin, Kafka, OpenCensus and OpenTelemetry compatible. It ingests batches in any of the mentioned formats, buffers them and then writes them to Azure, GCS, S3 or local disk. As such it is robust, cheap and easy to operate!

Tempo implements TraceQL, a traces-first query language inspired by LogQL and PromQL. This query language allows users to very precisely and easily select spans and jump directly to the spans fulfilling the specified conditions:

Getting Started

• Get started documentation
• Deployment Examples
  • Docker Compose
  • Helm
  • Jsonnet

Further Reading

To learn more about Tempo, consult the following documents & talks:

• New in Grafana Tempo 2.0: Apache Parquet as the default storage format, support for TraceQL
• Get to know TraceQL: A powerful new query language for distributed tracing

Getting Help

If you have any questions or feedback regarding Tempo:

• Grafana Labs hosts a forum for Tempo. This is a great place to post questions and search for answers.
• Ask a question on the Tempo Slack channel.
• File an issue for bugs, issues and feature suggestions.
• UI issues should be filed with Grafana.

OpenTelemetry

Tempo's receiver layer, wire format and storage format are all based directly on standards and code established by OpenTelemetry. We support open standards at Grafana!

Check out the Integration Guides to see examples of OpenTelemetry instrumentation with Tempo.

Other Components

tempo-vulture

tempo-vulture is Tempo's bird themed consistency checking tool. It writes traces to Tempo and then queries them back in a variety of ways.

tempo-cli

tempo-cli is the place to put any utility functionality related to Tempo. See Documentation for more info.

License

Grafana Tempo is distributed under AGPL-3.0-only. For Apache-2.0 exceptions, see LICENSING.md.

tempo's Issues

Compress Objects

All traces referenced by a single index record could be compressed.

This would occur when a block is completed in the ingester. Currently it "downsamples" according to a configurable value and only stores one index record for N objects. During this process we could also compress this group of N objects using gzip or snappy or something.

Metric total objects per tenant

Add this metric.

Fix Flaky Test TestJanitorCleanupOrder

TestJanitorCleanupOrder was commented out b/c it's flaky. Data as reported from the fileinfo objects is often not what is expected.

Make vendor introduces a lot of changes

Consider adding vendor/ to .gitignore?

Compactor seems to pick bad blocks

Configured with a 60m maxCompactionRange the compactor is choosing blocks over 1 hour apart:

level=info ts=2020-02-27T14:56:33.118545538Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:ef16962b-85b0-4d4c-96cf-25e4efbf775b MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 0 127 53 218 50 155 24 40] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:1114175}"
level=info ts=2020-02-27T14:56:33.215026831Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:3c2f9209-2767-44cb-8c40-d893f9c082f4 MinID:[0 0 0 0 0 0 0 0 0 11 85 40 191 206 203 8] MaxID:[0 0 0 0 0 0 0 0 4 184 161 121 117 202 81 175] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:426499}"
level=info ts=2020-02-27T14:56:33.309527672Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:34292379-20a3-4f12-bc98-f305da331c1b MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 0 8 56 75 22 24 15 158] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:1152174}"
level=info ts=2020-02-27T14:56:33.403106762Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:15d841e8-a4ec-43ae-adff-71b731686a5a MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 3 236 175 12 175 121 214 105] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:18:24.28298673 +0000 UTC TotalObjects:202623}"
level=info ts=2020-02-27T15:01:48.041123907Z caller=compactor.go:266 msg="writing compacted block" block="&{Version:v0 BlockID:1c2902f1-59e0-42b5-8efc-432e854ffdd5 MinID:[0 0 0 0 0 0 0 0 0 0 0 5 37 123 223 172] MaxID:[0 0 0 0 0 0 0 0 0 13 28 120 172 46 134 68] TenantID:fake StartTime:2020-02-26 18:26:48.053504759 +0000 UTC EndTime:2020-02-26 19:43:24.517317671 +0000 UTC TotalObjects:1447736}"

Improve Proto Generation

• Research and use gogoslick?
• Add code generation to Make
• Currently vend is being used to force pull in proto files necessary for building our proto. Consider using a submodule.

Improve Configuration Ergonomics

Cortex and Loki can be intimidating to configure. Running ./app --help dumps a huge number of options immediately on the operator. Research and improve this experience.

• Remove most configuration options from the CLI and require a config file?
• Build a meta configuration layer that chooses sensible defaults for Cortex dependencies simplifying frigg configuration?

Move to OpenTelemetry Proto

The distributor and frigg-query both have god-awful protobuf translation code. From opencensus proto -> otel proto and from otel proto -> jaeger proto.

Soon (tm) the otel collector will implement this code correctly at which point we should replace ours.

Research map alternatives

The ingester instance has a go map that needs to sustain 1000s of updates per second. After #5 is in place for benchmarking consider alternatives such as:

https://github.com/cornelk/hashmap

Benchmark uints instead of bytes for ID

Currently a byte slice is used everywhere for the ID. Benchmark using 2 uint64's (idHigh, idLow) instead of the byte slice to represent the 128 bit trace id.

Investigate Climbing GCS latency

GCS get latency climbs and partially resets on restart:

I believe this is the block list polling taking longer as there are more blocks, but the reset is weird.

Fix response codes from querier/ingester

Response codes/metrics are broken. Investigate and improve.

Some examples:

• Query success rate stays 100% on failed queries
• Querying a nonexistent trace id returns internal server error

Compress Indexes

The indexes are currently written to storage with no compression and assume a 16 byte (128 bit) key. Both of these things are bad.

Add compression. Drop the key length assumption.

Change single binary org id from "fake"

This was confusing to me when I first started on Loki/Cortex. Make it something better.

Finish Compactor Module

The compactor module kind of technically exists, but is not really independently configurable. Finish it.

Fix compaction metrics

Compaction Duration and Compaction Stop Duration seem capped at ~8 minutes. I just saw a 2 million traces get compacted over ~25 minutes and it was reported at 8.

Increase range of these histograms. Maybe 30 minutes?

Compaction Gives up on Life

After compacting all possible blocks the compactor just quits:

During this same time period metrics indicate that the maintenance cycle and compaction duration were hitting. Apparently no blocks were being returned from the blocksToCompact function?

Restarting the compactor got it working again.

Add S3 Support

Add a friggdb backend for S3.

Add Max MBs per block in the ingester

The headblock is cut when a certain time or number of traces ends up in the block:

i.headBlock.Length() >= maxTracesPerBlock || i.lastBlockCut.Add(maxBlockLifetime).Before(now)

Add Max size in MBs and consider removing max traces. This plays into #32 so you probably want to work with that guy.

Prune unnecessary etcd goroutines

tempo starts up etcd goroutines even when not using the etcd client. Prevent these from being started. Possibly requires a PR to Cortex.

goroutine 57 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0001c9b40)
	/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
	/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x80

goroutine 106 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0001f0a60)
	/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
	/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x80

goroutine 109 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0001f0c00)
	/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:173 +0x3a3
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
	/home/joe/dev/grafana/frigg/vendor/go.etcd.io/etcd/pkg/logutil/merge_logger.go:91 +0x80

Make Readme Not Terrible

• Stuff
• Contributing guidelines
• Basic documentation
• Airplane mode/Getting started guide

Pass a Logger/MetricsFactory to TempoDB

The database code needs a way to log and publish metrics. We need to decide if it will be tightly integrated with the rest of tempo and just use the same logger as everything else. The other option being to pass these utilities as params.

Organize storage/access around a page

The current backend design hits a number of limitations as the size of the index and bloom filters continue to grow due to compaction. Reorganize around the idea of page access to allow for bloom/index/objects to grow larger as time goes on. Research and determine a good page size that is cache friendly and easy to work with.

One of the big concerns is partial updates. Partial reads are possible by using GET with the Content-Range header, but uncertain if partial updates are possible with the PATCH method. This will not matter to ingestion as it will cut relatively small blocks, but compaction will struggle as blocks become larger and larger.

• Bloom

  • Not straightforward at all. Unsure how to request/work with a partial bloom filters. Here are some ideas on partition bloom filters that are not directly applicable, but may be revealing:
    https://news.ycombinator.com/item?id=22010586
    https://github.com/nehbit/aether/blob/master/aether-core/aether/services/rollingbloom/rollingbloom.go

• Index

  • Currently the write/read paths are designed to pull/push an entire index at once. Split the index into n same sized "pages". We can then use the roughly linear distribution of trace ids to make guesses about which pages to pull first. This will greatly increase the max size of a block.

• Objects

  • This one is straightforward. We already index sets of objects per index record. It is currently based on a configurable value. We could change this to be max MB per index record instead of max traces.

https://en.wikipedia.org/wiki/External_memory_algorithm
https://en.wikipedia.org/wiki/Cache-oblivious_algorithm

If the compactor runs out of disk space it silently fails

currently the compactor has 0 disk space and is repeatedly logging this:

level=info ts=2020-02-27T14:06:47.382590293Z caller=compactor.go:144 msg="beginning compaction"
level=info ts=2020-02-27T14:06:47.382624782Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:71d77440-e5ae-46ae-9b61-a70f2414582b MinID:[0 0 0 0 0 0 0 0 0 0 0 10 189 238 6 54] MaxID:[0 0 0 0 0 0 0 0 127 254 241 7 100 74 22 76] TenantID:fake StartTime:2020-02-27 12:20:11.712776024 +0000 UTC EndTime:2020-02-27 12:33:06.386303856 +0000 UTC TotalObjects:103121}"
level=info ts=2020-02-27T14:06:47.430287526Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:bec15b74-6a7c-491d-89ee-fba9921a5a59 MinID:[0 0 0 0 0 0 0 0 0 0 0 31 65 182 242 69] MaxID:[0 0 0 0 0 0 0 0 127 253 53 140 68 27 44 94] TenantID:fake StartTime:2020-02-27 12:33:12.575160991 +0000 UTC EndTime:2020-02-27 12:46:36.583367246 +0000 UTC TotalObjects:103564}"
level=info ts=2020-02-27T14:06:47.469475667Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:fba54b79-0602-4428-85d8-59e1be48688f MinID:[0 0 0 0 0 0 0 0 0 0 0 11 214 90 88 248] MaxID:[0 0 0 0 0 0 0 0 127 248 235 225 79 137 18 27] TenantID:fake StartTime:2020-02-27 12:46:42.612485912 +0000 UTC EndTime:2020-02-27 13:00:06.413352755 +0000 UTC TotalObjects:103573}"
level=info ts=2020-02-27T14:06:47.519901468Z caller=compactor.go:151 msg="compacting block" block="&{Version:v0 BlockID:7fadee40-0bfa-40d1-ab8f-d89d95ddb8f0 MinID:[0 0 0 0 0 0 0 0 0 0 0 140 62 163 208 249] MaxID:[0 0 0 0 0 0 0 0 127 254 196 124 171 249 176 73] TenantID:fake StartTime:2020-02-27 13:00:13.094668004 +0000 UTC EndTime:2020-02-27 13:13:06.398800446 +0000 UTC TotalObjects:104019}"
level=info ts=2020-02-27T14:06:48.009173722Z caller=compactor.go:140 msg="compaction complete"

Add tests for Distributor routeRequest

This method is critical and untested and relatively easily testable. Add tests.

Improve Blocks to Compact Method

blocksToCompact currently just finds the first 4 blocks that fall within the compaction range. This will not be sufficient for HA mode, multiple ingesters, etc.

At least add:

• Maximum result block size to prevent attempting to compact enormous blocks
• Test for id overlap. This will prevent compaction on blocks that do not overlap and do not provide much compaction benefit.

Add WAL replay visibility

Log and metric WAL replay.

Fix ingestion path replication

We may have gotten too excited during the great butchering of Loki and removed more than was intended. In the distributor there is a method routeRequest that breaks a PushRequest up by traceid and forwards it to the appropriate ingesters based on tokens/replication factor. After this the distributor sends

Unfortunately it lost the ability to check for concensus on write. Readd this. Look at the replicationSet.Do() method from Cortex b/c I think it does this already.

Refactor Blocks Interface

There are ton of methods on the block objects. They are complicated and weird. Tighten things up.

• timewritten/blockwrotesuccessfully
• writeInfo

Duplicate Traces created on Ingester crash/restart

The ingester is designed to keep completed/flushed blocks for a configurable time period after it has been flushed. This time period exists to allow the queriers to pull new blocklists successfully before an ingester "forgets" a trace as well as to provide faster lookups on recent traces.

However, these completed blocks are stored in the wal folder. If an ingester restarts it will replay all of these traces locally and re-flush them to the backend.

Propagate context through Compactor interfaces

Propagate Context appropriately and add tracing support.

Read this: https://blog.golang.org/context and do non-dumb things.

Document lifecycle policies necessary to not leave partial blocks

In different scenarios partial blocks can be stranded with neither a meta.json nor a meta.compacted.json. The compactor should also act as a janitor and delete these blocks from the backend.

Improve blocklist polling

gcs block lists are currently polled serially. some improvements

1. poll concurrently. try to reuse the queue worker pool for this
2. poll for new blocks every n seconds, but only update old blocks every m seconds (m >> n)

2 is harder to pull off and less important. I think just implementing #1 for now will be fine.

Create CI

Add CI. At the minimum it should:

• Run tests
• Lint
• Display Code coverage
• Run and display the results of holistic benchmarks

Research and Add an In memory Cache

Add a configurable in memory cache to the /friggdb/backend/cache package.

Blocklist and Bloom should be cached in memory. Hold off on Object. I would love to see GroupCache work, but Ristretto is also an option.

Refactor compactor into own module

Move compactor to its own module and make sure all backend compactors implement backend.Compactor interface

Search Live Traces

Right now a trace has to make it to the headblock before it's searchable. Search the live traces.

Frigg won't come back up after OOM

If a wal replay fails just kill the block?

$ kc logs frigg-6b75444f9f-j87jk frigg
level=info ts=2020-02-20T00:24:52.765814165Z caller=config.go:130 msg=initialising module=server
level=info ts=2020-02-20T00:24:52.766143557Z caller=server.go:133 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2020-02-20T00:24:52.766349136Z caller=config.go:130 msg=initialising module=overrides
level=info ts=2020-02-20T00:24:52.767610908Z caller=config.go:130 msg=initialising module=store
level=info ts=2020-02-20T00:24:52.773444742Z caller=config.go:130 msg=initialising module=ring
level=info ts=2020-02-20T00:24:52.773584847Z caller=config.go:130 msg=initialising module=querier
level=info ts=2020-02-20T00:24:52.773694734Z caller=config.go:130 msg=initialising module=ingester
level=info ts=2020-02-20T00:24:52.774212314Z caller=client.go:212 msg="value is nil" key=collectors/ring index=1
level=info ts=2020-02-20T00:24:52.774324387Z caller=lifecycler.go:471 msg="not loading tokens from file, tokens file path is empty"
level=info ts=2020-02-20T00:24:52.774424424Z caller=lifecycler.go:495 msg="instance not found in ring, adding with no tokens" ring=ingester
level=info ts=2020-02-20T00:24:52.774707919Z caller=lifecycler.go:363 msg="auto-joining cluster after timeout" ring=ingester
level=error ts=2020-02-20T00:25:05.626133582Z caller=ingester.go:299 msg="error cleaning up headblock" error=null
level=error ts=2020-02-20T00:25:05.626265939Z caller=main.go:61 msg="error initialising Frigg" err="error initialising module: ingester: read 3261 but expected 14644"

Add CRCs to object block

To increase confidence in the data returned let's add a CRC to the object block.

See /friggdb/object.go

Find a way to test gcs backend

GCS backend is important and untested.

Additional CI

This issue exists to track additional CI we'd like/need implemented.

• push to dockerhub
• codecov.io
• Holistic benchmarks. See otel-collector/test-bed for a mature example.

Compaction Race Condition

blocksToCompact returns a slice of the blocklist:

return blocklist[cursor : cursorEnd+1] ...

It should copy these values into new slice with a different backing array which it passes to compact.

What's the deal with all these limits?

... and airline food?

Review the limits code that just came along with the Loki package. Make it make sense for frigg.

Make a Distributed Compactor

Need to think about the #32

Thanos compaction design doc:
https://docs.google.com/document/d/1U7SWi44pLsZrI2awWbz7gurn8kBHL7bmww3943mFC_w/edit?authuser=1#heading=h.v9f03rq2qq0b

Change the name of this thing to tempo

tempo is a good name. frigg is fun, but not :)

change it to tempo.

WAL replay triggers limits

The WAL replays so quickly it hits limits. This is bad. Fix.

Related to #42, #11

level=warn ts=2020-02-12T18:46:32.638180201Z caller=grpc_logging.go:37 method=/friggpb.Pusher/Push duration=87.8µs err="rpc error: code = Code(429) desc = per-user traces limit (local: 10000 global: 0 actual local: 10000) exceeded" msg="gRPC\n"

Create testing application

Create a loki-canary-like application that continuously tests tracing.

`make test` fails under go 1.14

-race catches a unsafe pointer conversion in godirwalk:

fatal error: checkptr: unsafe pointer conversion

goroutine 126 [running]:
runtime.throw(0x13ff65d, 0x23)
	/usr/local/go/src/runtime/panic.go:1112 +0x72 fp=0xc00006c860 sp=0xc00006c830 pc=0x463162
runtime.checkptrAlignment(0xc000299f18, 0x136c0e0, 0x1)
	/usr/local/go/src/runtime/checkptr.go:18 +0xb7 fp=0xc00006c890 sp=0xc00006c860 pc=0x434607
github.com/karrick/godirwalk.(*Scanner).Scan(0xc000326090, 0x3d)
	/home/joe/go/pkg/mod/github.com/karrick/[email protected]/scandir_unix.go:144 +0x11e fp=0xc00006c968 sp=0xc00006c890 pc=0x11c7d0e
github.com/karrick/godirwalk.walk(0xc0001ae300, 0xe, 0xc0000b0bd0, 0xc00006cc10, 0x0, 0x0)
	/home/joe/go/pkg/mod/github.com/karrick/[email protected]/walk.go:270 +0x40c fp=0xc00006cab8 sp=0xc00006c968 pc=0x11c92ec
github.com/karrick/godirwalk.Walk(0xc0001ae300, 0xe, 0xc00006cc10, 0xc000075240, 0x0)
	/home/joe/go/pkg/mod/github.com/karrick/[email protected]/walk.go:204 +0x36b fp=0xc00006cb88 sp=0xc00006cab8 pc=0x11c8b2b
github.com/grafana/frigg/friggdb/backend/cache.clean(0xc0001ae300, 0xe, 0x1, 0x1, 0x0, 0x0, 0x0)
	/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:84 +0x214 fp=0xc00006cc60 sp=0xc00006cb88 pc=0x11e6274
github.com/grafana/frigg/friggdb/backend/cache.TestJanitor(0xc0001b0900)
	/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache_test.go:109 +0xaf0 fp=0xc00006ced0 sp=0xc00006cc60 pc=0x11e80f0
testing.tRunner(0xc0001b0900, 0x1426ba0)
	/usr/local/go/src/testing/testing.go:992 +0x1ec fp=0xc00006cfd0 sp=0xc00006ced0 pc=0x5bd74c
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00006cfd8 sp=0xc00006cfd0 pc=0x4976b1
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1043 +0x661

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001b05a0, 0x13e7257, 0xb, 0x1426ba0, 0x1)
	/usr/local/go/src/testing/testing.go:1044 +0x699
testing.runTests.func1(0xc0001b05a0)
	/usr/local/go/src/testing/testing.go:1285 +0xa7
testing.tRunner(0xc0001b05a0, 0xc000241d50)
	/usr/local/go/src/testing/testing.go:992 +0x1ec
testing.runTests(0xc0001d2060, 0x1d37500, 0x2, 0x2, 0x0)
	/usr/local/go/src/testing/testing.go:1283 +0x528
testing.(*M).Run(0xc000190400, 0x0)
	/usr/local/go/src/testing/testing.go:1200 +0x300
main.main()
	_testmain.go:46 +0x224

goroutine 127 [select]:
github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor.func1(0xc00018d1a0)
	/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:56 +0x6f3
created by github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor
	/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:53 +0x4d

goroutine 69 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0003007e0)
	/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x4b2
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
	/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0xfc

goroutine 125 [select]:
github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor.func1(0xc00018cff0)
	/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:56 +0x6f3
created by github.com/grafana/frigg/friggdb/backend/cache.(*reader).startJanitor
	/home/joe/dev/grafana/frigg/friggdb/backend/cache/disk_cache.go:53 +0x4d

goroutine 32 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000183820)
	/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x4b2
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
	/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0xfc

goroutine 77 [chan receive]:
go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc000300a00)
	/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:173 +0x4b2
created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger
	/home/joe/go/pkg/mod/go.etcd.io/[email protected]/pkg/logutil/merge_logger.go:91 +0xfc
****

Handle Growing Index Sizes

As blocks are compacted the index sizes grow larger. Currently the codebase is not built to deal with this at all. Two implementation options:

• Currently the index is only readable as one big block. Add the ability to read partial indices. This impacts querying and compaction.
• Dynamically generate the number of records per index entry stored instead of using a configured value to help reduce the size of the index.

Improve querier concurrency between backend and ingesters

The querier first attempts to ask ingesters for a trace id. If an ingester is temporarily unavailable this will timeout. Add a tight timer or, even better, execute it concurrently with a backend query.