Dynamic Shape Support about xnnpack

[alankelly]

Hi Digant,

Dynamic input shapes are also interest us. Dynamic shapes are not supported today. The entire graph must be reinitialised if the shape of an input tensor changes. This is expensive as all weights must be re-packed and indirection buffers recalculated.

Support for dynamic input shapes are on our roadmap. This is very important to us too. However, unbounded dynamism will not be supported. A maximum value for each input dimension must be given and memory allocated using this. Some preliminary work has already be done. See here.

Using the lower level APIs to do this will be very slow so I strongly recommend not doing this. I am happy to speak with you to see if our plans will be useful to you.

[digantdesai]

Thanks for getting back to me. Glad to know that you are also interested in this feature.

Support for dynamic input shapes are on our roadmap. This is very important to us too. However, unbounded dynamism will not be supported. A maximum value for each input dimension must be given and memory allocated using this.

Upper-bounded, rank-specialized^ dynamism is something we are interested in at the moment as well. Do you have any timelines you can share for the roadmap?

Using the lower level APIs to do this will be very slow so I strongly recommend not doing this. I am happy to speak with you to see if our plans will be useful to you.

Agree, as I said earlier using lower level APIs is not the most desirable path for us as well. And I would be happy to sync with you on this and for the dynamic shape feature in general.

^ - Tensor rank is not dynamic.

[cbilgin]

Support for dynamic input shapes are on our roadmap.

Hi Alan, that's great news. We have a strong dependency to XNNPACK and this is one the main painpoints we have. I know a lot can chancge, but what's the rough timeline that you are thinking of for this?

[alankelly]

I am quite hesitant to give timelines publicly as we are extremely short staffed now. This should change soon so I will keep you updated as to our plans