hdparm missing about ceph-container

[Ulexus]

hdparm is not likely to run inside a container, anyway. It's likely going to need privileged mode and access to the /dev tree, at a minimum.

Where are you encountering the error?

[leseb]

I believe you saw this during the OSD startup, it usually tries to disable the write back cache of the journal device before opening it.
It's a warning not an error if I remember correctly.

Can you show the logs?

[hookenz]

No I lost it. You're right it was more of a warning and just noticed it during the startup/init process somewhere.

Actually I was wanting to pre-initialise the ceph to use an SSD journal device rather than having a journal file or journal on the same device. I have managed to get that to work using ceph-disk prepare.
Then mounting the drives, then starting ceph-osd.

[leseb]

OK great, I'm closing this. Feel free to re-open.
I think we need to see if we can improve the journal support.

[Ulexus]

@hookenz I do exactly this on most of my nodes: 1 SSD journal for several OSDs. These containers will work perfectly if you simply mount your journal to /var/lib/ceph/osd/journal (the default journal location, if it exists).

I run my OSD containers with -v /var/lib/ceph/osd:/var/lib/ceph/osd and have my SSD journal mounted to /var/lib/ceph/osd/journal. Each OSD will be automatically assigned a separate directory in that directory, so things should just work.

[Ulexus]

I just noticed that this functionality is not described in the documentation; updating now.

[hookenz]

So you use a journal file for each OSD instead of a journal partition?

I read somewhere about there being less overhead for a journal partition so I went that route. But I'm passing -v /dev:/dev to docker to make the disk available. I think I also need --privileged.
Your method avoids all that I think.
If I go the direction you recommend I'd have to create 2 journal mounts. It's a little more setup but I think I like your approach.

By the way, what filesystem are you initialising your journal drives with?

My configuration is 6 spinners and 2 SSD. So 3 spinners per SSD.

Oh and by the way, you're right about it coming through correctly when you pass /var/lib/ceph/osd. I was passing in /var/lib/ceph and then the filesystem was showing up as tmpfs. I haven't looked again to see if this is true. Maybe I did something else wrong. Anyway, it's working good.

[Ulexus]

I use all SSDs in my setup and all btrfs filesystems (journal and OSD). I used to use the parallel journaling support with btrfs, but somewhere along the line I quit doing so (I don't recall the reason, but parallel journaling requires --privileged, which I am not running).

I do recall hearing that the direct block device journals are better, but I haven't done any testing to say.

I think it's a reasonable trade (since disks are necessarily tied to host systems) to have the host system manage the mountpoints, so we just export /var/lib/ceph/osd for the container.