Reduced performance due to load balancer isolating hot tablets to a subset of shards about scylladb

[tgrabiec]

This has potential to cause regressions compared to vnodes, so I don't think it should be enterprise-only.

[mykaul]

If we've agreed it should go to 6.0, please set the milestone as such. Otherwise, I reckon 6.1.

[mykaul]

@bhalevy - who can work on this for 6.0?

[avikivity]

IMO important, very easy to regress in performance due to this. Also relatively easy to fix (adjust load balancer tablet selection algorithm).

[nyh]

@tgrabiec the solution you propose does not solve the general problem of hot tablets, but rather the more specific problem of hot tables, but I agree that the problem of hot tables is the more important case, so we should indeed do the fix you propose.

By the way, DynamoDB solved the problem of hot tablets (which they call "partitions") by:

1. Detecting them and splitting them to smaller tablets.
2. At some point this no longer helps (e.g., imagine that someone sends a huge number of requests to the same row), so they have a limit of 3000 RCUs and 1000 WCUs that a single partition will accept. So it cannot dominate the node it runs on.

[bhalevy]

@bhalevy - who can work on this for 6.0?

@tgrabiec, and he first needs to finish #16594
On top of that, we can statically balance tablets across shards in thee 6.0 scope (and it's a close call).

Post 6.0, I can see dynamic, heat-based balancing of tablets to disperse the load across nodes and across shards

[dani-tweig]

@tgrabiec, and he first needs to finish #16594

Assigning Tomek
@bhalevy , is there someone else who can do it? I think Tomek's plate is pretty full

[bhalevy]

@mykaul / @dani-tweig IMO we should postpone this issue to 6.1. Please ack.

[mykaul]

@bhalevy - we'd still like to see it in 6.0 (after syncing with @dorlaor , @avikivity , @tzach ) - who can be assigned to this (other than Tomek) ?

[mykaul]

@bhalevy , @tgrabiec :
Can we use the shuffle API as a valid workaround here, and push this from the 6.0 release blocker list?

[tgrabiec]

@bhalevy , @tgrabiec : Can we use the shuffle API as a valid workaround here, and push this from the 6.0 release blocker list?

I don't think we can rely on shuffle, it's unpredictable, may be completely ineffective or even make the situation worse.

[tzach]

I don't think we can rely on shuffle, it's unpredictable, may be completely ineffective or even make the situation worse.

It might, but what is the chance(percent)?
If it is low enough, we can live with it till we have a proper solution.

[bhalevy]

I don't think we can rely on shuffle, it's unpredictable, may be completely ineffective or even make the situation worse.

It might, but what is the chance(percent)? If it is low enough, we can live with it till we have a proper solution.

For 6.0 we plan to have a simple mitigation which is to randomize tablet selection in tablet load balancing using a uniform distribution. This should result in a similar distribution of tablets per table on the destination node on average, with large enough number of tablets.

[bhalevy]

@tgrabiec is going to submit a PR destined for 6.0 with randomized picking of source tablets by the load balancer.
Then, we can continue to improve the selection algorithm for 6.1 and later.