Update protocol to use generic devices vs storage about agent

[mcastelino]

/cc @amshinde @egernst @sboeuf

[egernst]

@bergwolf @laijs @gnawux @WeiZhang555

[laijs]

• device types

Storage.Driver

• device identification (PCI, /dev/xyz, ...) as it shows up in the VM

Storage.Source and
Storage.Options (it is planed to be added. map[string]string).

[sameo]

@laijs

Storage.Driver
Storage.Source

We need to rename Storage to Device then.

[sameo]

We should also manage the devices directly in the agent and try not to rely on udev etc. This will allow us to use the agent as init.

Something along those lines:
clearcontainers/agent#180

[sameo]

We should also manage the devices directly in the agent and try not to rely on udev etc. This will allow us to use the agent as init.

@mcastelino btw, I figured out our agent is dynamically linked because libcontainer relies on cgo for nsexec.

[laijs]

@sameo Storage is more generate the Device.

device identification as it show up with the container (predicted name)

@mcastelino I will add a pr to add the name

[sameo]

@laijs a device can be a storage device or any other device, so device is more generic than storage.

[laijs]

@sameo I think device is one kind of storage among all kinds of storages. For example, I think tmpfs is a storage rather than a device. Are there any reference to tell what are the differences between them?

[sameo]

@laijs In the storage context, device would be the hardware backend for your storage. For tmpfs, I guess it would be the ram controller.

Let me roll the converstion back a little: You were suggesting to use the Storage structure for describing generic devices. So I said if that's what we want, we need to rename Storage to Device, because otherwise it would be really confusing: We might argue that a Storage mount point does not always map to a Device, but I don't think anyone would argue that a Device does not always map to a Storage mount point. NICs, GPUs, FPGAs, etc...none of those map to any kind of Storage so we will not use a structure called Storage to describe them.

[laijs]

@mcastelino @sameo #55 is changing the storage api.

[sboeuf]

@laijs I think #55 increase the complexity for no reason. We will always want to add a storage/device when creating a new container, and remove those when removing the same container. Same thing for the sandbox.
Moreover, this PR does not change the name Storage into Device. FWIW, I also truly think that Device is more generic than Storage. Keep in mind that we are basically trying to solve the case where someone wants to use --device or --volumes when creating a new container. I think that we can come up with a generic Device structure, so that we can handle those both flags properly.

[laijs]

If --device or --volumes is storage such dir/nfs/cifs/block/ceph/9pfs we can use this Storage to describe them. If we need to something to describe NICs, GPUs, FPGAs we would at it later. Or we can have both Storage and Device.

In container world, we say "Storage driver", "Storage plugin", "Container Storage Interfaces", we doesn't say "Device driver", "device storage interfaces".

[mcastelino]

@laijs we already support VFIO in clear containers and would need to continue to support that. --device is not always a storage device

We support

docker run -it --device=/dev/vfio/16 centos/tools bash

Where the device can be a QAT device or a network card. It can be any PCI device.

[sboeuf]

If we cannot agree on device vs storage, then we might need to introduce Device along with Storage. Both of those passed through the CreateContainer() API.
@laijs Please notice that if we do that and if we also go with this PR #55, we'll end up adding AddDevice and RemoveDevice. I think this will be a lot of different functions serving the same purpose of creating a new container with all its resources...

[bergwolf]

The purpose of adding AddStorage/RemoveStorage is to make storage a pod level resource and let container OCI reference them in the Mount.Source field. AddStorage and RemoveStorage actually needs to mount/umount the devices from sandbox point of view.

If other devices have similar needs (being referenced by several containers, needing extra setup), then we need AddDevice/RemoveDevice as well.

[bergwolf]

IMHO, it is better to have separate Storage and Device struct. If we do want to converge them, can we please group storage specific fields together into a pointer, and do the same for other devices as well?

This is because different types of devices might have different attributes and these are hard to structurally define. Take #64 for example, fstype/mountpoint/mount_option are all storage device specific and they are present in the struct of other types of devices. Even if we do define the device attributes in a generic map, it introduces a lot of trouble parsing them and the parsing is error prone.

So can we please consider something like following?

type Device struct {
        DeviceId      string
        Driver        string
        DriverOptions map[string]string
        Device        string
        Storage       *StorageDevice
        Interface     *InterfaceDevice
        GPU           *GPUDevice
        FPGA          *FPGADevice
        VFIO          *VFIODevice
        Common        *CommonDevice
}

I understand this is a bit too complex but it again proves the need for separating storage and device types. These types are easy to unite on their own. It saves a lot of trouble and complexity to separate them in data structures. What do you guys think?

[jodh-intel]

The gRPC protocol currently includes storage and device so can we close this now?

[jcvenegas]

We already have both device and storage so closing issue.