Support for custom ONNX optimized models about keybert

[MaartenGr]

This is currently not supported as Flair, which is the main method for using embedding models, uses hugginface transformers which is typically not meant for importing onnx structures in part due to their tokenization structure. Exporting to onnx is possible however.

[sujithjoseph]

This is available in sentencetransformers - https://github.com/UKPLab/sentence-transformers/blob/72da7293186700bff4fed8e81adcfe40a03be1d2/examples/onnx_inference/onnx_inference.ipynb . We can load an onnx model for inference. With a local path to onnx exported file, it could be added as one more backend . Did test it with tinyBERT exported onnx model, but didnt see much performance gain on a V100. yet to try on a T4 GPU.

[MaartenGr]

Thank you for the link! I did not realize it was already available in sentence-transformers. The main difficulty though will be loading in custom ONNX models that require some processing before it can be used in sentence-transformers.

[sujithjoseph]

I am thinking of adding it as a custom backend. i didnt see this before - https://maartengr.github.io/KeyBERT/guides/embeddings.html#custom-backend

[MaartenGr]

Yes, if you can create a custom backend that loads in using the sentence-transformer then that would solve your issue. It would also be nice if that backend could then be added to KeyBERT but that would also depend on dependencies and implementation.

[MaartenGr]

Since this issue has been a while without activity, I'll be closing it for now. However, if you are still experiencing the issue or want to discuss it further, let me know!

[sujithjoseph]

Sure, Will add a PR later to add this back to KeyBERT. -Sujith

…

On Mon, Oct 25, 2021 at 11:20 PM Maarten Grootendorst < ***@***.***> wrote: Since this issue has been a while without activity, I'll be closing it for now. However, if you are still experiencing the issue or want to discuss it further, let me know! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#48 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABC7JFUBPL5MN7KWBLXHNTDUIZJC5ANCNFSM5ABWWQNQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Matthieu-Tinycoaching]

Hi @MaartenGr @sujithjoseph did you manage to deal with ONNX backend?