BERT model implemented usiing TransformerEncoder returns all NaNs when running it torch==1.13.1 about aws-neuron-sdk

[jluntamazon]

Thank you for the great example! We were able to reproduce the problem.

The most likely cause of this issue is due to the exact operations that are lowered across different versions of torch. The compiler receives a different graph which may trigger different instructions to be executed. Some of these operations may cause very large (infinity) values to appear in one version of torch while these will not appear in another version. One way to fix this is to set the compiler to optimize for this type of model:

• --model-type=transformer - This options enables compiler optimizations for transformer models. This can avoid NaN values by performing more specialized methods which are less prone to numerical issues.
• (Optional) --enable-saturate-infinity - This option ensures that large values do not go to infinity. Infinity values can cause NaNs to be produced due to the underlying hardware operations, so this can avoid problems with large values.

I was able to produce non-NaN values by configuring the --model-type flag prior to compilation:

import os
os.environ['NEURON_CC_FLAGS'] = '--model-type=transformer'

[aws-rhsoln]

@sgaseretto , does the above recommendation solve the issue?

[sgaseretto]

Hi @aws-rhsoln, yes it solved the issue! Sorry for the late response, closing this issue now