Refactor OSD setup about ceph-container

[hookenz]

I might tie the bootstrap keys into the ceph-cli work. The way i now have the osds bootstapping is fast. I am pretty sure i can hotplug them.

[Ulexus]

@leseb I think it sounds like a good idea. The additional --privileged requirement makes me less sanguine, but it seems there is the choice of having the OSD fully bootstrapped (with the privileges that entails) or having to work something outside the container.

I'm also not sold on the use of ceph-disk; it's an enormous script with a lot of assumptions built in. I was fairly permanently put off of ceph-deploy thanks to those assumptions and complexity. On the other hand, it's maintained by the Ceph project, so it's probably less fragile than manually building the OSDs.

Originally, I preferred the approach of separation, but for more massive deployments than my own, a fully-bootstrapped approach seems more sensible.

You'll still need to be able to safely reference which block devices to use, and unless we open up ceph-disk, configuration-wise, this will be less flexible. I would definitely prefer to maintain a path for handing over a filesystem to the OSD, rather than have it all done for you, but I think that could be best done with better detection, rather than explicit environment variables, as is done currently.

Overall, I like it.

[leseb]

The prototype can be found in #78 here: https://github.com/ceph/ceph-docker/pull/78/files#diff-b9c25bf1154f3e0eb1449dd01310fbc3R187

[leseb]

I'm really in favour of using ceph-disk can (as your said) its maintained by the Ceph core developer. It is intended to best user experience, along with best practices.
What I don't like with the OSD directory path is that it requires a pre-configuration (it can be automated on the host though) where with ceph-disk you just pass a block device and that's it.

[Ulexus]

I'd like to make sure its options (filesystem and journal types) are exposed, but if you're confident in it, I'm fine with using it.

I certainly agree regarding the preconfiguration, but I do think that should still be an allowed setup. I don't think that's a mutually-exclusive feature; we just need to tighten up the detection mechanism. The OSD store has everything necessary in it in a nicely-readable format. We just need to use it.

[leseb]

I agree. Operators shouldn't be forced to use --privileged thus we need to propose different ways of setting up the OSD. At the moment I can think of two:

• keep the current format where we need to prepare the OSD disk, filesystem, mount, create and expose the OSD directory
• use ceph-disk