dynamic locality loadbalancer behaviour with hpa about istio

[ramaraochavali]

Discussed offline very briefly with @howardjohn .

@hzxuzhonghu WDYT?

[howardjohn]

How is "skewed" determined?

Say I have 3 zones. If I have 3/3/3 pods in each, is it "skewed"? I am only sending to 1/3 of the pods otherwise.

Is it 1/4/4? 1/1/7?

What if I am 3/3/3 but all the clients happen to be in one zone?

[ramaraochavali]

How is "skewed" determined?

It is determined based on skew_factor if it is set to 2, if any zone has > 2x pods compared to any other zone, it is considered skewed

What if I am 3/3/3 but all the clients happen to be in one zone?

If your 3 pods can handle all client requests happily - no change. If they can not handle and hpa is triggered , if hpa results in skew - we get in to disable locality mode

[howardjohn]

The thing that is confusing to me.

Why is 3/3/3 ok, but 3/6/0 we should start change?

From the local zone its the same - 3 pods (33%) get traffic

[ramaraochavali]

Why is 3/3/3 ok, but 3/6/0 we should start change?

That is a good point. The fact it went from even distribution(3/3/3/) assuming original distribution was good to 3/6/0 is a "hint" that tells us some thing changed via hpa. I do not know if we can compare the previous vs. current distribution to identify the skew. I know it is not perfect but my idea is to create this issue so that we can discuss and see if we can solve this.

[howardjohn]

I feel like this is not the responsibility of a load balancer TBH, but of the scheduler to schedule pods where they are required

[hzxuzhonghu]

I agree with john's point here. It is the scheduler not loadbalancer who should be in charge of this. LB is working as expected. Yes, sure it lacks the capability to be aware of the server load. There is a issues tracking this in envoy i think.

[ramaraochavali]

I agree with the point that it is schedulers responsibility. But given how HPA works, I am trying to see if Load Balancer can be intelligent to handle this case.

Yes, sure it lacks the capability to be aware of the server load.

It is not just server load but combination of server load + the zone in which it is scheduled. So similar to how we fallback to other regions when all endpoints are unhealthy, I think it would be good to have a mechanism in Load Balancer that spills over to other zones if the current zone's pods are overloaded (not just unhealthy). Ofcourse the solution I proposed just assumes "skew" as an indicator for overload which is not correct in all cases.

Can you point me to envoy issue if you have it handy?

[howardjohn]

The concern I have is I cannot come up with a reasonable algorithm for when we should starting spilling over due to skew that solves your use case, isn't stateful, and isn't just "round robin"..

[ramaraochavali]

I do not think switching to round robin would help unless we disable locality load balancer unless I am missing some thing in your proposal.

[hzxuzhonghu]

@ramaraochavali envoyproxy/envoy#6614 If you want more intelligent lb, this is the right requirement

[ramaraochavali]

I do not think it will work when locality load balancer is enabled. When locality load balancer is enabled, we pick nodes based on priority first and once priority is picked we apply the load balancer (least request, round robin, cost aggregated etc). Does not it pick nodes in the same zone and apply this cost logic?

[hzxuzhonghu]

Sure, locality load balancer collaborate with some other unimplemented algorithms could solve this case

[ramaraochavali]

I think the key is to disable locality load balancer or make it behave in away to spill over traffic to other zones when we detect some abnormality in pod scheduling/load if enabled. How can we correctly detect is the question

[ramaraochavali]

BTW, the skew above is similar to how k8s evaluates max_skew in PodTopologyConstraints. This is an interesting article on how even with PodTopologyConstraints can result in skew https://medium.com/wise-engineering/avoiding-kubernetes-pod-topology-spread-constraint-pitfalls-d369bb04689e - especially during scale down