When Attempting to loadSavedModel, I Encountered 'java.lang.Exception: Could Not Retrieve the SavedModelBundle + () about spark-nlp

[Alexwangziyu]

Hi @TerryLaw535

To import T5 models (fine-tuned or already on HF) you can use one of these 2 notebooks:

• TensorFlow: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/HuggingFace%20in%20Spark%20NLP%20-%20T5.ipynb
• ONNX: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/onnx/HuggingFace_ONNX_in_Spark_NLP_T5.ipynb

Depending on which one you have saved (exported) your T5 model, you can follow the instructions to import your model into Spark NLP. (I personally recommend first trying ONNX, it has a faster inference time)

hi maziyarpanahi, thanks for your resource, however, I tried different transformers, tensorflow versions but none of them worked, basically there are all kinds of errors. And the code doesn't work on colab correctly either. Could you please check it out? Thank you so much!

[maziyarpanahi]

Hi @TerryLaw535

To import T5 models (fine-tuned or already on HF) you can use one of these 2 notebooks:

• TensorFlow: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/HuggingFace%20in%20Spark%20NLP%20-%20T5.ipynb
• ONNX: https://github.com/JohnSnowLabs/spark-nlp/blob/master/examples/python/transformers/onnx/HuggingFace_ONNX_in_Spark_NLP_T5.ipynb

Depending on which one you have saved (exported) your T5 model, you can follow the instructions to import your model into Spark NLP. (I personally recommend first trying ONNX, it has a faster inference time)

[TerryLaw535]

Hi @maziyarpanahi

Thank you so much for your reply. I tried ONNX and it showed that [ONNXRuntimeError] : 1 : FAIL : Loading the model from onnx_models/google-t5/t5-small/decoder_model_merged.onnx failed:/onnxruntime_src/onnxruntime/core/graph/model.cc:179 onnxruntime::Model::Model(onnx::ModelProto&&, the onnxruntime::PathString&, the onnxruntime::IOnnxRuntimeOpSchemaRegistryList*, the onnxruntime::logging::Logger&, the onnxruntime::ModelOptions&) Unsupported model IR version: 10, max supported IR version: 9.

I also tried the TensorFlow method. I completely followed the instructions except for setting tensorflow == 2.8, since this version is too old and no longer available. I tried the code:
try:
model = T5ExportModel.from_pretrained(MODEL_NAME)
except:
model = T5ExportModel.from_pretrained(MODEL_NAME, from_pt=True)

model.export(EXPORT_PATH, use_cache=True)

and it report:
TypeError: in user code:

File "/tmp/ipykernel_70993/430537825.py", line 50, in decoder_init_serving  *
    logits = self.shared(sequence_output, mode="linear")
File "/home/weichengyu/.local/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler  **
    raise e.with_traceback(filtered_tb) from None

TypeError: Embedding.call() got an unexpected keyword argument 'mode'

Do you know how to deal with these problems? I think these problems are all related to version issues.

Thank you!

[maziyarpanahi]

Hi @Alexwangziyu

I will assign someone to have a look at the TensorFlow to Spark NLP, something might have changed in Hugging Face. For the T5, I will ask someone to make a quick test to see if the notebooks are up to date.