Quite largely increased latency with weights/neff separated about aws-neuron-sdk

[jluntamazon]

Hi @JingyaHuang,

When weights are not inlined, there are some effects that can reduce performance:

• The compiler must respect the shape and data type of the input parameters provided:
  • This means that it does not have control to downcast weights to the given --auto-cast data type. If you do not -explicitly downcast the model weights yourself this can mean that the underlying model may consume an fp32 weight and then be required to downcast it at runtime to fp16/bf16 for subsequent auto-casted compute.
  • When the compiler is provided the weights it has the ability to transform the layout and shape for optimal weight loading. This helps with improving the data load sizes. Currently there is no good way to mitigate this penalty when using non-inlined weights, but there is work underway on our side to improve this.
  • The compiler can propagate changes to or completely eliminate weights when they are known. A good example of this is when using a lower triangular causal mask. When the compiler sees this inlined to the graph, it will attempt to avoid compute on half of the masked matrix. Since the compiler cannot know ahead of time if a mask will be a lower triangular when weights are not inlined, it is forced to do the full compute. To improve on this issue, it possible to write torch operations that do not represent mask as a constant tensor. In an upcoming release this will be improved by always inlining non-Parameter constants.
• There are some known performance penalties in the runtime when using non-inlined weights from passing all of the weight tensors to the underlying model object. We have seen that depending on how large a model is, this can account for a significant overhead. This will be fixed in a coming release.
• Currently when separating weights, all constants are pulled out as weights. In a coming release this will be restricted to module nn.Parameters only which may improve performance of inline masking/tensors/scalars.

We can look into this specific model and see which of the above effects is causing poor performance.