Can we get the OSD_ID variable back ? about ceph-container

[Ulexus]

I'll add the default hostname extraction, but I don't really see why you need to pass the OSD_ID. Right now, it derives the osd id from the path. If you want your osd container to only run a single osd, just pass it a single path.

Can you elaborate on your difficulties?

[tangzhankun]

hi @Ulexus ,
Thanks for quick reply. Here is my understanding for current ceph/osd entrypoint:
It scans the /var/lib/ceph/osd and get "ceph-${OSDID}" from the directory name. If I don't make stupid mistake. This container may get OSD 0 and 1 and start two daemons. right? But I want to let it start one container even if the mount directory has multiple "ceph-*" in it.
And here are the commands (just for hint) I use to start multiple ceph/osd using below command:
docker exec mon0 ceph osd create
(I got an OSD_ID 0)
mkdir /cehp_config_root/var/lib/ceph/osd/ceph-0
mount disk
docker run -v /ceph_config_root/var/lib/ceph/osd:/var/lib/ceph/osd -v /ceph_config_root/etc/ceph:/etc/ceph ceph/osd
(it will start osd with OSD_ID 0)
Then I want to start another osd
docker exec mon0 ceph osd create
(I got an OSD_ID 1)
mkdir /cehp_config_root/var/lib/ceph/osd/ceph-1
mount disk
docker run -v /ceph_config_root/var/lib/ceph/osd:/var/lib/ceph/osd -v /ceph_config_root/etc/ceph:/etc/ceph ceph/osd
This time it will start two deamons which is not what I want.
I can put the ceph-${OSDID} directory to another place to workaround this. But I want to put them all in a same parent directory. So this is OSD_ID needed which avoid scan the directorys.

[Ulexus]

The trick is to limit what you export to the osd. If you want the daemon to only run for osd 0, in your scenario, you would run:

docker run -v /ceph_config_root/var/lib/ceph/osd/ceph-0:/var/lib/ceph/osd/ceph-0 -v /ceph_config_root/etc/ceph:/etc/ceph ceph/osd

I still fail to see a good reason to run the osd's in separate containers, however. That you can does not mean you should. In particular, you run the risk of hitting the various port exhaustion problems, such as referenced by issue #19. While the present system is a workaround that you are bypassing, since we don't actually know the real dynamics, I would advise you do not run multiple osd containers on a single box unless you have a very compelling reason to do so.

[tangzhankun]

hi @Ulexus,
Thanks for the trick. I thought OSDID is more straightforward for users. I'll use the trick then.
We are developing a ceph-mesos project which aim to scale ceph cluster in Mesos production environment with ceph dockers.
The first reason why I put OSDs in separate containers is that I want good isolation between OSDs. If one OSD failed ( which is expected in production scenario ), it won't affect other OSDs. This is a benefit that container technology bring to us I think. And the second reason is that it's easy to manage each OSD container's status in ceph-mesos if we separate them.
For issue19, I checked the workarounds that you provides. Then I want to choose Macvlan to assign different container with different IPs in ceph-mesos's next feature. During my investigation, the ceph cluster seems work fine under Macvlan. I'll do further testing later.
Thanks again. If you have interests, I am very glad we work together , discuss current issues more and make ceph better with container. :)

[Ulexus]

@tangzhankun Thanks; I'm definitely interested to hear your results, and your reasoning sounds good to me. I mainly wanted to make sure you weren't making the decision for the wrong reason.

The problem with network overlays and Docker, for me, has always been the dynamic nature of their IP assignment. I've been toying with the idea of making OSDs completely self-contained by the container. That is, they register a new OSD when the container starts, and remove it when the container stops. That eliminates the problem of bouncing IPs, but it leaves open the possibility of a lot of extra churn in the cluster, particularly on systems like CoreOS which update automatically on a fairly regular schedule. How are you handling potentially-changing container IPs with Ceph?

[tangzhankun]

Hi @Ulexus ,
In my opinion, the changing IP of container for ceph is not the key issue. The key issue is that Ceph needs to know the OSD's location (rack, host, etc) to distribute replica to different host to ensure data availability. If we are using different IP for different OSD, Ceph may put three replicas in same host. So a doable solution is that we tell Ceph the location of OSDs ( in ceph.conf or use same hostname, haven't try this ). A self-contained OSD container is a good idea, but still faces the problem mentioned above.

For now, I think we cannot say that ceph in container can be used to production environment. There maybe a lot of gaps we need to fill.

[Ulexus]

@tangzhankun Aha! That's an incredibly important feature of Ceph, and we provide no facility to specify that metadata at all. I've been doing it manually, which is not a problem on my relatively small cluster, but that should absolutely be supported in the OSD creation and (probably) daemon startup.

Please feel free to open another issue or PR on that. We definitely need to support it.

[tangzhankun]

hi @Ulexus,
I tried launch OSD with the physical hostname which it resides in. It works. This can avoid the potential issue related to data availability. So we got a workaround which make ceph docker closer to production ceph with Macvlan.
By the way, why no update on issue19? The best solution I think is still launching multiple OSD with "--net=host."

[leseb]

The CRUSH location is done here: https://github.com/ceph/ceph-docker/blob/master/daemon/entrypoint.sh#L438
However we don't expose anything to make it more flexible. We could easily add a LOCATION variable that you can use like this:

LOCATION='root=default datacenter=dc1 room=1 rack=10 host=$(hostname)'

[Ulexus]

@tangzhankun From your description... please tell me you aren't NFS-mounting a bunch of remote disks to use as the basis for a Ceph cluster.

@leseb I like the free-form style of denotation with just LOCATION, but I believe the tags are actually a restricted set, yes?

# types (from http://ceph.com/docs/master/rados/operations/crush-map/)
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

I think we should offer a free-form CRUSH_LOCATION for "experts," but we should also offer each of the types explicitly (ex. CRUSH_RACK).

[leseb]

@Ulexus answering on #121

[tangzhankun]

hi @Ulexus ,
No. I don't consider NFS. I want the container to bind-mount a directory which use the local host's disk. Just like ceph usually do. I am investigating on solution for this dedicated disk feature in Ceph-Mesos.
This is what I am thinking at present:

1. user inputs available disk name like "sdb"
2. we got "sdb", parted it to /dev/sdb1
3. mkfs /dev/sdb1 and mount it to the directory which will be bind-mounted to a OSD
4. start the osd
   Any suggestions?

hi @leseb
Thanks for the tip. I haven't use ceph/daemon. Seems it's more powerful. But for LOCATION, I have a question on #121 there.