How to scale the read path? about victoriametrics

[hagen1778]

it takes most of the work time to receive data from the vmstorage-s (since there are many of them)

vmselect receives data from vmstorages in parallel, so many vmstorages shouldn't delay data retrieval. It should be opposite, the more vmstorages you have the faster they process the search query and respond with data.

and aggregate data on vmselect-s.

If aggregation on vmselect takes significant amount of time this is where vertical scaling should help. Please note, vmselect does query processing in parallel fashion, running a separate worker per each available CPU. These workers are responsinble for unpacking data received from vmstorages and performing concurrent calculations before the final merge.

[Sinketsu]

Ok, we will try to increase our vmstorage nodes count too. Thanks!

[hagen1778]

Do I understand correctly that if it is not possible to vertically scale vmselects, then it remains only to introduce sequential processing through recording rules /stream aggregation, which will cause its inconveniences (the names of the metrics will not match their original names) and delays?

It is correct. vmselect can't shard query execution among other vmselects. Hence, it can be scaled only vertically for resource-intensive queries.
The scaling logic is the following:

1. If you have intensive read queries - prefer a smaller number of vmselects with high resource capacity
2. If you have many light-weight queries - prefer a bigger number of vmselects with light resource capacity

These two approaches could be mixed. For example, alerting and recording rules are usually considered as lightweight load, since they usually select short time intervals. So the ruler (vmalert) could be configured to "talk to" a fleet of many lightweight queries.
If you have some heavy Grafana dashboards or other queriers, configure them to "talk to" a fleet of smaller number of heavy vmselects which would crunch big amounts of data faster.

Or maybe there are other ways to improve the cluster's read performance? I will be glad for any advice/thoughts.

I'd recommend starting with https://docs.victoriametrics.com/troubleshooting/#slow-queries.
Recommendations from the doc and query tracing could help to understand where the bottleneck is. It could be vmselect's compute resource, but it also could be vmstorage disk IO, vmstorage CPU, network between vmstorage and vmselect. In this case, scaling vmselects will have no effect on query latency.

[Sinketsu]

1. If you have intensive read queries - prefer a smaller number of vmselects with high resource capacity
2. If you have many light-weight queries - prefer a bigger number of vmselects with light resource capacity

These two approaches could be mixed. For example, alerting and recording rules are usually considered as lightweight load, since they usually select short time intervals. So the ruler (vmalert) could be configured to "talk to" a fleet of many lightweight queries. If you have some heavy Grafana dashboards or other queriers, configure them to "talk to" a fleet of smaller number of heavy vmselects which would crunch big amounts of data faster.

An interesting idea, we will try to divide them, thank you!

Or maybe there are other ways to improve the cluster's read performance? I will be glad for any advice/thoughts.

I'd recommend starting with https://docs.victoriametrics.com/troubleshooting/#slow-queries. Recommendations from the doc and query tracing could help to understand where the bottleneck is. It could be vmselect's compute resource, but it also could be vmstorage disk IO, vmstorage CPU, network between vmstorage and vmselect. In this case, scaling vmselects will have no effect on query latency.

We looked at it. In our case (judging by the traces), it takes most of the work time to receive data from the vmstorage-s (since there are many of them) and aggregate data on vmselect-s.

[Sinketsu]

it takes most of the work time to receive data from the vmstorage-s (since there are many of them)

vmselect receives data from vmstorages in parallel, so many vmstorages shouldn't delay data retrieval. It should be opposite, the more vmstorages you have the faster they process the search query and respond with data.

I mean a lot of data blocks to retreive for query. Now we have 5 vmstorage servers. Do you have any recommendations on how many hosts it is optimal to have, depending on the amount of data? Maybe we should increase their number for better performance...

and aggregate data on vmselect-s.

If aggregation on vmselect takes significant amount of time this is where vertical scaling should help. Please note, vmselect does query processing in parallel fashion, running a separate worker per each available CPU. These workers are responsinble for unpacking data received from vmstorages and performing concurrent calculations before the final merge.

Yes, I understand that, but we can no longer scale vmselect vertically(

[hagen1778]

Do you have any recommendations on how many hosts it is optimal to have, depending on the amount of data?

It doesn't depend on amount of data, but rather on the bottlenecks during query processing. For example, you have 5 storage nodes, each has local disk with 100MB/s read throughput. Then one query (in the best case) can utilize 500MB/s read speed. If you add 5 more vmstorage nodes, then one query would be able to utilize 1000MB/s.

In short, scaling number for vmstorage nodes is similar to scaling the number of concurrent workers - it is always beneficial. Unless the vmselect is the bottleneck and can't process the received data fast enough.

Scaling vmstorage nodes makes sense up to 30-50 nodes. After that, the fragility of network starts to kick-in increasing query latency because of higher probability of network lag.

[hagen1778]

Closing question as answered. Feel free to re-open or create a new issue if you still have questions.