The choice of the split size for splitAndFuse about deepspeed-mii

[tohtana]

Hi @frankxyy,
Dynamic SplitFuse splits a long prompt to fit it to the remaining capacity of the specified forward size, rather than evenly splitting. This approach optimizes the efficiency when you receive many requests.

Maybe it is better split long input into chunks of sizes in decreasing order?

I'm not sure if I fully understand your suggestion. Can you elaborate on it?

[frankxyy]

@tohtana Hi, I misunderstood the approach of splitAndFuse, in fact the splited size is determined by the remaining capacity of forward... Thank you for your remind.

[frankxyy]

@tohtana Hi, I rethink about this issue and have some new opinions. I find that as the sequence length increases, the compute intensity of the same forward size differs(the q_len remains the same, but kv_len increases). So the splitting to the same forward sizes may not split to the same compute intensity?

[tohtana]

@frankxyy Yes, the amount of computation varies depending on lengths of kv.
We have a target forward size to balance the efficiency and latency but I'm not sure how much difference the varying kv lengths cause.