Errors when loading saved onnx files about fastt5

[itaim]

Yes it does! Thank you very much!

[sidsharma72]

I was facing the same issue - the resolution lies in the order of the quantized model paths.
It should be encoder, decoder, init-decoder.

[Ki6an]

what's your onnxruntime version and does the models folder have all the 3 quantized models?

[itaim]

The models folder has the 3 quantized model files. I am not sure what is my onnxruntime version. How can I find it?

[Ki6an]

pip3 show onnxruntime
if you are using the latest version of fastt5 you should have onnxruntime v1.7.0

[itaim]

Name: onnxruntime
Version: 1.7.0
Summary: ONNX Runtime is a runtime accelerator for Machine Learning models
Home-page: https://onnxruntime.ai
Author: Microsoft Corporation
Author-email: [email protected]
License: MIT License
Location: /Users/itaimarks/Code/email-cleaner/.venv/lib/python3.7/site-packages
Requires: numpy, protobuf
Required-by: fastt5

[Ki6an]

can you provide the reproducible code? also, are you running it on a docker containerized environment?

[itaim]

Yes running in containerized environment, and generating the onnx files in the Dockerfile.

from fastT5 import (
    generate_onnx_representation,
    quantize,
)
import shutil
import os


model_path = "models/email_cleaner"
onnx_model_paths = generate_onnx_representation(model_path)
quant_model_paths = quantize(onnx_model_paths)
print(quant_model_paths)
for file in quant_model_paths:
    shutil.move(file, model_path + "/" + os.path.basename(file))

for file in onnx_model_paths:
    os.remove(file)

def init_cleaner_model(model_path):
    try:
        start = time.time()
        app.logger.info(f"Loading Cleaner T5 model from: {model_path}")
        tokenizer = AutoTokenizer.from_pretrained(model_path)
        os = platform.platform()
        if os.startswith("Darwin"):
            app.logger.info(f"OS: {os}, Using vanilla T5 model")
            model = T5ForConditionalGeneration.from_pretrained(model_path)
        else:
            app.logger.info(f"OS: {os}, Using FastT5 model")
            quant_model_paths = glob.glob(model_path + "/*-quantized.onnx")
            model_sessions = get_onnx_runtime_sessions(quant_model_paths)
            model = OnnxT5(model_path, model_sessions)

        app.logger.info(
            f"Loading cleaner model and tokenizer took {time.time() - start} seconds"
        )
        return model, tokenizer
    except Exception as ex:
        app.logger.error(ex, exc_info=True)
        return None

[Ki6an]

I tried fastt5 on a Linux container for simple QG rest API with flask, and it worked fine. I've linked the drive files for the project you can take a look if you are interested. meanwhile, I'll look into your code.

[Ki6an]

I think the order of quantized models has changed while using glob.glob() to get the model path. You might be feeding only 2 inputs (of encoder) to init decoder (it needs 3 input values).

Make sure to provide the quantized models path in this order to get_onnx_runtime_sessions.

LMK if this solves your issue.