Current and future status of rbd-volume about ceph-container

[leseb]

Unfortunately I've never got the chance to play with it, so I don't really know.
However I don't see any reasons why you wouldn't give it a try :).
@Ulexus will definitely answer your question but as far as I know he is currently on Holidays so you might have to wait a bit :).

[hookenz]

Correct me if I'm wrong but as I understand this issue,

In some earlier docker versions --volumes-from also worked from containers with network nfs shares etc. But then it was removed (mount namespaces added or modified) and --volumes-from some-other-container containing remote mounts no longer worked.

Therefore rbd-volume will never work while docker has this limitation. A pity that! Because of the direction docker seems to be taking in this regard, I think rbd-volume should now be removed as it's never going to work and it appears unlikely it ever will. Instead they are coming up with an alternative method (storage drivers?) but these haven't yet implemented for ceph I think.

Which leaves only the rbd device mounted from the host or from the container directly. On CoreOS mounting on the host is difficult so here is my best alternative which you may find useful - rbdmap (Full credit to Laurent Barbe for the initial implementation and details of how it works).

[Ulexus]

Thanks to a number of patches from @dmitryint (rbd-related PRs), it works in special cases, but I wouldn't design a system around it.

At best, it is a stop-gap hack to work around lack of RBD support for Docker volumes. I looked at implementing that a year or so ago, and at that time, I didn't see a good place to hook in RBD support to the volume code. I'm hoping with the new storage driver, that will change... but up to this point, I believe that only affects the container root, not volumes.

For now, it is (as @hookenz just said), best to mount the RBD volumes to the host and bind-mount them into the container, which is what I do.

[hookenz]

Hmm, that docker rbd-volume fix @dmitryint came up with looks interesting! I'll have to experiment with that. But, am I wrong in thinking that shouldn't work actually?

[Ulexus]

As I'm combing through my vacation email, I see a reference to this on the docker-dev list:

https://github.com/calavera/dkvolume

Docker has volume plugins, now. Yay! That's definitely the way to go for rbd-volume, not the current implementation.

[hookenz]

@Ulexus - If I get some time over the next week I'll write a volume plugin for RBD.

[h0tbird]

Checkout this docker-volume-ceph-rbd I have just discovered :-)

[hookenz]

@h0tbird - Sweet! Does it work!?

[hookenz]

Of course, this is all well and good, but on CoreOS you're kindof stuck with the stable (not experimental branch) of docker. Which means these efforts can't be used by most people yet. Still, it's an exciting development. Looking forward to seeing this in the mainstream.

[h0tbird]

@hookenz I don't know if it works I guess we have to try it ;-)
Thank you all for the info! I think I have enough to close the issue now.

[Ulexus]

A note: Docker version 1.8.0 was just release with official support for volume plugins.

[h0tbird]

So far on the radar:

• https://github.com/contiv/volplugin
• https://github.com/yp-engineering/rbd-docker-plugin
• https://github.com/AcalephStorage/docker-volume-ceph-rbd

[leseb]

I'm planning on testing this soon.

[h0tbird]

@leseb great! I have already started to test one.
Check this issue I have just opened: yp-engineering/rbd-docker-plugin#1

[leseb]

@h0tbird just looked at the various repositories, setting up all these things is not as easy as I thought it'd be :). It looks like someone has started an interesting initiative here: contiv-experimental/volplugin#10