Lease expiration v.s. scale up trigger about fenzo

[spodila]

Nice work. Glad to note Fenzo was useful for your framework.

Did you use the TaskScheduler.Builder.withDelayAutoscaleUpBySecs(secs) to make the scale up less aggressive? We introduced that for very similar reasons. You should not need to introduce the new delay you are referring to after the last offer expiration. The JavaDocs for that builder method has additional explanation on why and what to set the delay to. I realized the JavaDocs links weren't updated to the latest ones, I will do that shortly. You can see the java docs in the code here.

I notice in your code that you introduced a similar delay to scale down outside Fenzo. You can use TaskScheduler.Builder.withDelayAutoscaleDownBySecs(secs) for that. When Fenzo gives a call back for scale down, it disables those hosts internally before the call back, for a time equivalent to the configured cool down seconds for the AutoScale rule. If you further delay the scale down, the hosts for which Fenzo called the scale down action will not be used until the cooldown period expires. So effectively, you have fewer servers to assign resources from.

I suggest you use the built in auto scale delay feature.

Also, since you are using the auto scaling feature, it would be useful for you to use bin packing as the fitness calculator (e.g., BinPackingFitnessCalculators.cpuMemNetworkBinPacker) when building the scheduler.

Let me know if you find these don't work for you. My understanding is that they should.

[spodila]

@yanglei99 The java docs are now updated. You can see the scale up delay doc here.

Let me know if you need any further info.

[yanglei99]

@spodila, Thanks! Will follow your suggestions and let you know the result.

[spodila]

@yanglei99 Do you still need this issue open?

[spodila]

@yanglei99 I am going to close this issue since there doesn't seem to be an open question after the last discussion. Do open a new one if you have a follow up. Thank you.