Concurrent requests counter at vmstorage holds requests, aborted up to 15 minutes ago about victoriametrics

[Haleygo]

Hi @dglushenok and sorry for late response!
I can't seem to reproduce this issue as well, all the connections are closed as expected after data blocks uploaded.
Could you share the vm_concurrent_queries results on vmselect(both vmselect-native and vmselect-http) during that time? Is there any relevant logs on those vmstorage or vmselect?

Then I see increase of vm_vmselect_concurrent_requests_current counter for 15 minutes on all vmstorage instances of zonal cluster which reported the error (but no increase on zonal cluster, which was skipped due to search.skipSlowReplicas). The same increase I see on vmselect zonal cluster, which runs in native mode.

If you remove --search.skipSlowReplicas flag from vmselect-http, does the issue happens for all the storage nodes?

[dglushenok]

Could you share the vm_concurrent_queries results on vmselect(both vmselect-native and vmselect-http) during that time?

On the screenshot below, vmselect-multicloud-sts-npe is vmselect-http, vmselect-zonal-sts-prod is vmselect-native:

Is there any relevant logs on those vmstorage or vmselect?

I found no logs related to hanged requests.

If you remove --search.skipSlowReplicas flag from vmselect-http, does the issue happens for all the storage nodes?

With disabled search.skipSlowReplicas I was able to reproduce the issue, but noticed, that not every query, aborted by search.maxSamplesPerQuery, causes the problem. Then I noticed the same behavior with enabled search.skipSlowReplicas - the same query, aborted by search.maxSamplesPerQuery, may or may not cause hanged request.

Answering your question - disabling search.skipSlowReplicas can cause the issue on all storage nodes, but after firing two queries one by one I end up with one hanged request on first vmstorage subcluster and two hanged requests on second: