BERT Server Demo - Fails to create kernel due to invalid NEFF about aws-neuron-sdk

[keithachorn-intel]

@spring01 - I applied the code change provided and re-attempted compilation. The error code resulted after 45+ minutes on a c5.4xlarge instance:

Traceback (most recent call last):
  File "bert_model.py", line 246, in <module>
    main()
  File "bert_model.py", line 113, in main
    raise AttributeError('Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly (`pip install neuron-cc` to install neuron-cc).')
AttributeError: Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly (`pip install neuron-cc` to install neuron-cc).

So I am now left knowing that the error occurred during compilation rather than when the bert_server.py was launched. However, there is no indication why the compilation failed. The input saved model - created using Appendix 1 - appears adequate. And neuron-cc is properly installed.

Should this require a new issue ticket or is this one sufficient?

[micwade-aws]

Hi, keithachorn-intel.

We were able to reproduce and are actively debugging the issue. Thanks for making us aware of this issue. We will update back on fix progress. No need for a new ticket - this one is sufficient.

Mike

[micwade-aws]

Hi, keithachorn-intel.

Please update to the latest version of Neuron SDK software. The latest release has a specific change to address this escape and we’ve plugged the hole with additional testing. Thanks again for letting us know and please reopen this issue if you find it’s still not working.

Mike

[yugoren]

I believe I have the same issue even though I brought up my instance yesterday and ran all the updates today again. I'm working on an inf1.6xlarge instance for all steps and I have a working neuron-cc installed:

(aws_neuron_tensorflow_p36) ubuntu@ip-172-31-70-98:~/NeuronSDK/src/examples/tensorflow/bert_demo$ neuron-cc --version
Neuron Compiler version 1.0.4680.0+5844509397

HWM version 1.0.602.0-5843326937
NEFF version 0.6
TVM version 1.0.1219.0+5844423152
NumPy version 1.17.2
MXNet not available
TF version 1.15.0

I already have the model stored and running bert_model.py results in the following:

(aws_neuron_tensorflow_p36) ubuntu@ip-172-31-70-98:~/NeuronSDK/src/examples/tensorflow/bert_demo$ python bert_model.py --input_saved_model $BERT_LARGE_SAVED_MODEL --output_saved_model ./bert-saved-model-neuron --crude_gelu
WARNING:tensorflow:From bert_model.py:144: fuse (from tensorflow.python.neuron.python.fuse) is deprecated and will be removed in a future version.
Instructions for updating:
Please refer to AWS documentation on Neuron integrated TensorFlow 2.0.
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:
  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
  * https://github.com/tensorflow/addons
  * https://github.com/tensorflow/io (for I/O related ops)
If you depend on functionality not listed there, please file an issue.

2020-01-30 22:16:04.706512: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
2020-01-30 22:16:04.728128: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2999995000 Hz
2020-01-30 22:16:04.729283: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5617b1b6e730 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-30 22:16:04.729307: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:From /home/ubuntu/anaconda3/envs/aws_neuron_tensorflow_p36/lib/python3.6/site-packages/tensorflow_core/contrib/predictor/saved_model_predictor.py:153: load (from tensorflow.python.saved_model.loader_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.loader.load or tf.compat.v1.saved_model.load. There will be a new function for importing SavedModels in Tensorflow 2.0.
WARNING:tensorflow:From bert_model.py:32: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From bert_model.py:35: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From bert_model.py:214: The name tf.rsqrt is deprecated. Please use tf.math.rsqrt instead.

WARNING:tensorflow:From bert_model.py:89: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.

Traceback (most recent call last):
  File "bert_model.py", line 245, in <module>
    main()
  File "bert_model.py", line 112, in main
    raise AttributeError('Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly (`pip install neuron-cc` to install neuron-cc).')
AttributeError: Neuron executable (neff) is empty. Please check neuron-cc is installed and working properly (`pip install neuron-cc` to install neuron-cc).

[keithachorn-intel]

@yugoren - If you're using a model you previously compiled, try starting again with the [uncompiled]] Saved Model and re-compile it. Doing this fixed the issue for me.

[yugoren]

@keithachorn-intel I tried re-running the entire exercise from scratch, no changes. One change I did was to modify the fine-tuning part to train only for 0.1 epoch rather than 3 (since it takes unnecessarily long; I just want to test the latency numbers for BERT with inf1 instance for now), and I think it's a valid change looking at the implementation here.