How does tf-encrypted integrate with TF Serving? about tf-encrypted

[ianlivingstone]

@morgangiraud once you have completed your investigation, could you post your notes here?

[morgangiraud]

Multiple questions arise from this:

Why TF serving can be interesting for us

It specifically handle the usecase of load once and serve multiple request which is mandatory for our usage due to the overhead of network optimisation done by TF when loading a distributed graph.

It's also made for model versioning, updates and rollback which can be nice in the long run.

Does our models can be loaded in TF serving

As long as we keep using Tensorflow functions under the hood to build our graph, we are fine.

TF serving provides signatures for regression, calssification and a generic prediction. Those are used to create a gRPC API and/or a REST API. Those APIs consume JSON, convert and route it for your graph inputs automatically. As long as the graph is only containing native C++ functions, one can load its graph using TF Serving.

Note that we can build new C++ ops and add them to a custom TF build, that will be used to build a custom TF Serving.

Can we do distributed Tensorflow Serving

There is currently no use cases for Google to implement it there is no obvious reason why distributing computation for inference would speed up the said inference.

But, hopefully, at least one guy wanted to do it and he has been kind enough to post his exploration on StackOverflow here.

Based on his notes, I gave it a go and see if I could save a distributed model on localhost with a cluster and use it in RF Serving out of the box: spent 1h30 building using bazel to end up erroring...

First good point:

• all the needed bits are here to do distributed computation using TensorFlow Serving and TF clusters.

Second good point:

• When defining a model and save it. Devices are saved by default, meaning that as long as you define your cluster and keep it online. TF Serving is just using the graph which handles the distributed computation.

Potential bad points:

• This is not officialy supported so might run into multiple problems
  • Model versioning might be broken.
  • What happens if the master node stops?
  • What happens if anhy stops actually?
  • Error handling maybe?

TODO

Actually seeing if it works out of the box or you have to build from scratch. I'll retry inside a docker env soon

Interesting feedback from zendesk

https://medium.com/zendesk-engineering/how-zendesk-serves-tensorflow-models-in-production-751ee22f0f4b

1 year old though, things have improved since