Consider defering endpoint plug to container start about libnetwork

[mrjana]

@tomwilkie Thanks for bringing it up. With the CNM design we wanted to decouple network management from container lifecycle management. IOW, we wanted the network resources like endpoints to have independent lifetime compared to containers themselves so even when a container restarts when it comes back and uses the same endpoint it is guaranteed to get the same set of resources. From what we have seen in the docker user community they would prefer much to see the same resources provided to the container every time it restarts.

This all works in a nice way with the current api. If we introduce the Plug/Unplug methods then we tie in endpoint resources to container lifecycle and different drivers will start behaving differently for these methods and the user experience is going to differ based on those. So in effect the time window between endpoint creation and container start (which would not be that much) is to be considered as the same case as the time when container is down i.e container is down but endpoint is available.

Does that make sense?

[tomwilkie]

@mrjana Thanks for getting back to me! I can understand that you want networks & endpoint instances to have a separate lifecycle to containers; I agree with this.

Conflating the lifecycle of network & endpoint instances with the resources associated with these instances is risky; you are mandating a particular behaviour for network providers beneath your nice, abstract interface. You are risking introducing limits beyond what is obvious - for instance, assuming a network has a limited number of available IPs (most I've seen do) you are forcing the total number of containers to match these limits (not the total number of running containers).

I guess a potential reason for perceiving this method as preferred by the docker user community is due to the lack of built in name resolution - not a problem with the Weave Network (and others). If our plugin is forced to allocate IPs and names on container creation, without an appropriate hook for container start and stop, our distributed DNS will be forced to contain stale entries or work around this plugin interface.

As you recognise, introducing a Plug method (and returning a SandboxInfo) would leave the decision about the lifecycle of resources to the plugin; I think you need to accept and encourage different plugins implementing the interface differently.

I don't think it's safe to assume the time window between endpoint creation and container start will be small; it's a perfectly reasonable usecase to have lots of non-running containers sitting around. Users will be more used to resources being bound to containers when they are run - this is how DHCP and volumes attachments (LUNs) work with physical and virtual machines.

If the concern is that allocating the resources could be an expensive operation, the cost of which you do not wish to incur on container start, then my proposed modification still allows for plugins to cache and reuse these resources to amortise the cost of a Plug operation.

TLDR different network plugins will want to manage their resources differently; this should be embraced and encouraged, and we shouldn't assume at this stage to be able to envisage all potential implementations.

[mapuri]

Is it more a question of when CreateEndpoint() hook is called? I am thinking if it is called before the container starts (when the container's network namespace is created) it shall be good enough. Likewise DeleteEndpoint() can be called just before the container's network namespace is deleted.

Or do we really need two separate hooks one for container create and one for start and likewise for stop and delete. I can't think of much that a network driver can do as part of container create and delete when the namespace doesn't even exists.

[dave-tucker]

This was taken care of as part of the libnetwork/docker integration and is documented in https://github.com/docker/libnetwork/blob/master/docs/design.md