Support downloading models from S3/Blob or mounts for TensorRT about kserve

[rakelkar]

#117

[ellistarn]

I've been thinking about the model storage problem. Re-downloading from S3/GCS is somewhat expensive. It would be kind of nice to shared a shared model cache that lives somewhere, maybe as a shared mountable volume.

We could then have some process for loading the model to some local volume and then pointing for all model servers to read from there. We could encapsulate TensorRT details onto that volume. Thoughts?

[yuzisun]

Looks like there is complication of syncing from S3/GCS model storage, TensorRT does polling and then adds/removes the model versions(https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-master-branch-guide/docs/model_repository.html#modifying-the-model-repository).

[rakelkar]

Well if we were to copy, then that feature would be disabled as the server would only poll the local unchanging copy... for production this seems like the right thing to do. For interactive this is probably okay. In general we need to think about consistentency of experience across different model types vs exposing the native behaviors of servers. Thoughts?

…

On Sun, Jun 2, 2019 at 9:05 PM Dan Sun ***@***.***> wrote: Looks like there is complication of syncing from S3/GCS model storage, TensorRT does polling and then adds/removes the model versions( https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-master-branch-guide/docs/model_repository.html#modifying-the-model-repository ). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#137>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2CTBICW3NUEZEQD46ENGTPYSKAVANCNFSM4HSB7P4A> .

-- Rakesh Kelkar

[yuzisun]

Definitely agreed, we can disable this at this stage, however there is real prod use case for continuous training and active learning where this feature can be useful and manual canary is not an option, the rollout risk is managed somewhere else in the train-serve loop.

[rakelkar]

I was thinking of downloading in an init container and exposing a mount to the tensorRT container, however as noted in #129 we need to figure out how to enhance or workaround knative here...

[ellistarn]

Why do you need to expose a mount. If you download in the init container, the pod shares a disk, right?

[rakelkar]

pod shares a disk, right

They may share the disk on the node if that is what you mean, but each container has its own filesystem so you would still need mount support

[rakelkar]

closed via #148