ceph / ceph-container

Docker files and images to run Ceph in containers

Ceph - a scalable distributed storage system

See https://ceph.com/ for current information about Ceph.

Status

Contributing Code

Most of Ceph is dual-licensed under the LGPL version 2.1 or 3.0. Some miscellaneous code is either public domain or licensed under a BSD-style license.

The Ceph documentation is licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

Some headers included in the ceph/ceph repository are licensed under the GPL. See the file COPYING for a full inventory of licenses by file.

All code contributions must include a valid "Signed-off-by" line. See the file SubmittingPatches.rst for details on this and instructions on how to generate and submit patches.

Assignment of copyright is not required to contribute code. Code is contributed under the terms of the applicable license.

Checking out the source

Clone the ceph/ceph repository from github by running the following command on a system that has git installed:

git clone [email protected]:ceph/ceph

Alternatively, if you are not a github user, you should run the following command on a system that has git installed:

git clone https://github.com/ceph/ceph.git

When the ceph/ceph repository has been cloned to your system, run the following commands to move into the cloned ceph/ceph repository and to check out the git submodules associated with it:

cd ceph
git submodule update --init --recursive --progress

Build Prerequisites

section last updated 27 Jul 2023

Make sure that curl is installed. The Debian and Ubuntu apt command is provided here, but if you use a system with a different package manager, then you must use whatever command is the proper counterpart of this one:

apt install curl

Install Debian or RPM package dependencies by running the following command:

./install-deps.sh

Install the python3-routes package:

apt install python3-routes

Building Ceph

These instructions are meant for developers who are compiling the code for development and testing. To build binaries that are suitable for installation we recommend that you build .deb or .rpm packages, or refer to ceph.spec.in or debian/rules to see which configuration options are specified for production builds.

To build Ceph, make sure that you are in the top-level ceph directory that contains do_cmake.sh and CONTRIBUTING.rst and run the following commands:

./do_cmake.sh
cd build
ninja

do_cmake.sh by default creates a "debug build" of Ceph, which can be up to five times slower than a non-debug build. Pass -DCMAKE_BUILD_TYPE=RelWithDebInfo to do_cmake.sh to create a non-debug build.

Ninja is the buildsystem used by the Ceph project to build test builds. The number of jobs used by ninja is derived from the number of CPU cores of the building host if unspecified. Use the -j option to limit the job number if the build jobs are running out of memory. If you attempt to run ninja and receive a message that reads g++: fatal error: Killed signal terminated program cc1plus, then you have run out of memory. Using the -j option with an argument appropriate to the hardware on which the ninja command is run is expected to result in a successful build. For example, to limit the job number to 3, run the command ninja -j 3. On average, each ninja job run in parallel needs approximately 2.5 GiB of RAM.

This documentation assumes that your build directory is a subdirectory of the ceph.git checkout. If the build directory is located elsewhere, point CEPH_GIT_DIR to the correct path of the checkout. Additional CMake args can be specified by setting ARGS before invoking do_cmake.sh. See cmake options for more details. For example:

ARGS="-DCMAKE_C_COMPILER=gcc-7" ./do_cmake.sh

To build only certain targets, run a command of the following form:

ninja [target name]

To install:

ninja install

CMake Options

The -D flag can be used with cmake to speed up the process of building Ceph and to customize the build.

Building without RADOS Gateway

The RADOS Gateway is built by default. To build Ceph without the RADOS Gateway, run a command of the following form:

cmake -DWITH_RADOSGW=OFF [path to top-level ceph directory]

Building with debugging and arbitrary dependency locations

Run a command of the following form to build Ceph with debugging and alternate locations for some external dependencies:

cmake -DCMAKE_INSTALL_PREFIX=/opt/ceph -DCMAKE_C_FLAGS="-Og -g3 -gdwarf-4" \
..

Ceph has several bundled dependencies such as Boost, RocksDB and Arrow. By default, cmake builds these bundled dependencies from source instead of using libraries that are already installed on the system. You can opt to use these system libraries, as long as they meet Ceph's version requirements. To use system libraries, use cmake options like WITH_SYSTEM_BOOST, as in the following example:

cmake -DWITH_SYSTEM_BOOST=ON [...]

To view an exhaustive list of -D options, invoke cmake -LH:

cmake -LH

Preserving diagnostic colors

If you pipe ninja to less and would like to preserve the diagnostic colors in the output in order to make errors and warnings more legible, run the following command:

cmake -DDIAGNOSTICS_COLOR=always ...

The above command works only with supported compilers.

The diagnostic colors will be visible when the following command is run:

ninja | less -R

Other available values for DIAGNOSTICS_COLOR are auto (default) and never.

Building a source tarball

To build a complete source tarball with everything needed to build from source and/or build a (deb or rpm) package, run

./make-dist

This will create a tarball like ceph-$version.tar.bz2 from git. (Ensure that any changes you want to include in your working directory are committed to git.)

Running a test cluster

From the ceph/ directory, run the following commands to launch a test Ceph cluster:

cd build
ninja vstart        # builds just enough to run vstart
../src/vstart.sh --debug --new -x --localhost --bluestore
./bin/ceph -s

Most Ceph commands are available in the bin/ directory. For example:

./bin/rbd create foo --size 1000
./bin/rados -p foo bench 30 write

To shut down the test cluster, run the following command from the build/ directory:

../src/stop.sh

Use the sysvinit script to start or stop individual daemons:

./bin/init-ceph restart osd.0
./bin/init-ceph stop

Running unit tests

To build and run all tests (in parallel using all processors), use ctest:

cd build
ninja
ctest -j$(nproc)

(Note: Many targets built from src/test are not run using ctest. Targets starting with "unittest" are run in ninja check and thus can be run with ctest. Targets starting with "ceph_test" can not, and should be run by hand.)

When failures occur, look in build/Testing/Temporary for logs.

To build and run all tests and their dependencies without other unnecessary targets in Ceph:

cd build
ninja check -j$(nproc)

To run an individual test manually, run ctest with -R (regex matching):

ctest -R [regex matching test name(s)]

(Note: ctest does not build the test it's running or the dependencies needed to run it)

To run an individual test manually and see all the tests output, run ctest with the -V (verbose) flag:

ctest -V -R [regex matching test name(s)]

To run tests manually and run the jobs in parallel, run ctest with the -j flag:

ctest -j [number of jobs]

There are many other flags you can give ctest for better control over manual test execution. To view these options run:

man ctest

Building the Documentation

Prerequisites

The list of package dependencies for building the documentation can be found in doc_deps.deb.txt:

sudo apt-get install `cat doc_deps.deb.txt`

Building the Documentation

To build the documentation, ensure that you are in the top-level /ceph directory, and execute the build script. For example:

admin/build-doc

Reporting Issues

To report an issue and view existing issues, please visit https://tracker.ceph.com/projects/ceph.

ceph-container's Issues

Monmap management

I was speaking with @TelanJ earlier about what happens in a dynamic environment (like K8s) with multiple Mons. Cases where a Mon being started with a different hostname and/or IP.

For a quick test, 7 mon containers were started, then 2 stopped and another 2 started. The monmap contained 9 mons of which 2 were down. This would have been my expectation but I don't think it's an ideal option for live environments.

It's not easy to know if a mon is temporarily offline of it's been migrated to a different hostname or IP. Ceph also makes it very hard as updating a monmap means all mons need to be restarted (http://ceph.com/docs/master/rados/operations/add-or-rm-mons).

To throw out a few incomplete ideas:

• Manual - Entry command/arg that cleans up missing mons (doesn't handle stop/injection/starting of all mons)
• Use the KV store for tracking active mons. Removing old mons when starting a new mon. Use confd to track changed keys and stop/start the mon inside the container.
• Give mons fixed IDs (similar to OSDs). If the ID is the same then check the IP (in monmap dump?) and remove/add as needed

None are really complete solutions since there needs to be ways of ensuring quorum, injecting the updated monmaps and regenerating the ceph.conf. Any thoughts on the best way to handle the monmap?

Create new images with tags for ceph version

We currently ship firefly as latest tag, we should move to giant and add a tag for firefly.

Next steps:

• Create firefly branch in Github
• Create firefly tag referencing that branch on Docker Hub
• Update each FROM statements in Dockerfiles for downstream components in the firefly branch to reference ceph/base:firefly
• Create firefly tags for each downstream component on Docker Hub
• Create giant tags for each component on Docker Hub

Support CRUSH locations

Spawned by discussion from #113

CRUSH location types http://ceph.com/docs/master/rados/operations/crush-map/:

type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

I think we should offer a free-form CRUSH_LOCATION for "experts," but we should also offer each of the types explicitly (ex. CRUSH_RACK), whose purpose and use should be more easily-grasped, though such may be more verbose.

Multiple instances of radosgw fills conntrack table in seconds

I am testing this docker container, thank you, but I have a problem, when I run multiple instances, suddenly the conntrack table gets filled.
e.g. I run one instance, sometimes 2, and nothing wrong, also:

cat /proc/sys/net/netfilter/nf_conntrack_count

gives me less than 100
but after one or two more instances, nf_conntrack_count increases very quickly, and after a few seconds it reaches 65k, I tried increasing it to 256k but it gets filled in seconds too.

Rebuild ceph/base

Since wget was missing, the repo's key is not present.

Can't create an automated Docker Registry build without access to the github Ceph org

I was going to set up the Docker Registry automated builds for these Dockerfiles, but I cannot do so without access to the Github Ceph org. (I do have access to the Docker Registry Ceph org.)

Forbidden error when using ceph-demo and accessing radosgw through API

Once ceph-demo is started, radosgw responds

curl localhost
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

In order to access radosgw using API I need to create a user. Following
http://ceph.com/docs/master/radosgw/config/
we can issue

docker exec cd30ab09156 radosgw-admin user create --uid="testuser" --display-name="First User"

and then use Python code like this one

import boto
import boto.s3.connection
conn = boto.connect_s3(
    aws_access_key_id='3HYE82O7DGS49LKIY4WJ',
    aws_secret_access_key='pARGxDCQ+D3fS+s6EQjeCGWLnEhMWdbncXeB\/hQu',
    host='localhost',    
    is_secure=False,
    calling_format = boto.s3.connection.OrdinaryCallingFormat(),)

    bucket = conn.create_bucket('my-new-bucket')

Unfortunately I get

    bucket = conn.create_bucket('my-new-bucket')
  File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 590, in create_bucket
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

What else am I missing?
As far as I can tell ceph-demo is no different from ceph-rgw in terms rgw default behaviour.

Share the docker command and IP settings to build a cluster?

Would someone could nicely sharing how to build up a simple cluster with docker command and the IP specified method? Thx in advance.

I'm confusing about MON_IP, how can I get it before run it up to go Ceph-config?

Possible to reduce image sizes by using different base image?

It might be possible to reduce the size of the docker images by using a different base image such as alpine.

Add ceph-rest-api in ceph/demo image

ceph-rest-api is one of the components of ceph which is really useful for monitoring. I think we can add it in the all-in-one ceph/demo image. The command to run ceph-rest-api looks like this.

ceph-rest-api -n client.admin

Can we get the OSD_ID variable back ?

Hi,
I am trying to launch multiple ceph/osd in one host with macvlan( different IP for issue 19), each on a separate disk. This scenario is more like a production needs. But I found ceph/osd is not suitable at present:

1. It should accept OSD_ID. Disappointed that it was removed.
2. It should get it's hostname in container by "hostname -s" like what the ceph/mon do. Because it seems difficult for us to know the container hostname before it start.
   Can someone implement this? Thanks in advance.

Current and future status of rbd-volume

I am opening this issue to ask about the current status of rbd-volume and what is planned for it in the future. I am trying to figure out a nice pattern to deliver RBDs to my user's containers. Seems that moby/moby#8854 will be the long term solution but meanwhile I need another approach.

So, why is rbd-volume tagged as NON-FUNCTIONAL? Should I give it a try?

getting ceph demo running

I'm interested in testing how ceph can be accessed through S3 API or, failing that, some other API. Any help would be much appreciated. It seems that the ceph demo starts RGW so using it could be an easy start. What are the steps to get the ceph-demo running? Specifically, am I right assuming that ceph/config has to be run (or config created) before running ceph/demo?

Unfortunately for me ceph/config hangs on "Configuration is locked by another host. Waiting." and ceph/demo immediately exits (no config?) although I have been able to run both with limited success in the past. This is what I'm trying to run now (etcd is running on the host itself)

docker run -e ETCDCTL_PEERS=http://172.16.X.Y:4001 -e MON_IP=172.16.X.Y \
-e MON_NAME=mymon -e CLUSTER=testing -v /etc/ceph:/etc/ceph ceph/config

docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=172.16.X.Y \
-e CEPH_NETWORK=172.16.0.0/16 ceph/demo

ceph/daemon totally broken

for example, if run MDS:

sudo docker run -d --net=host \
-v /var/lib/ceph/:/var/lib/ceph/ \
-v /etc/ceph:/etc/ceph \
-e CEPHFS_CREATE=1 \
ceph/daemon mds

in entrypoint.sh kviator runs with --kvstore=none argument, what causes error: Backend storage not supported yet, please choose another one

Radosgw container migrate to another server

As we know. Radosgw container is the gateway of Ceph on CoreOS Cluster,the Radosgw container
sometimes migrate to another server . How to solve this issues?

Ceph REST API container

I will add an image as soon as this lands: ceph/ceph#4191

config doesn't actually work for bootstrapping

While it works if a configuration already exists, the script bombs out if the key doesn't exist

Refactor OSD setup

Hey,

The way the OSDs are currently setup is not straightforward at all. It's really confusing how pieces interact with each other. Thus I'm proposing to rework a bit how we bootstrap the OSD.

In #78 I started integrating bootstrap-keys, so we don't need to have the ceph.admin key inside all the containers and more importantly on all the hosts running the containers.
So yes there is a prefetching operation that needs to collect the appropriate keys.
The operation will create the following entries:

/var/lib/ceph/
|-- bootstrap-mds
|   `-- ceph.keyring
|-- bootstrap-osd
|   `-- ceph.keyring
|-- bootstrap-rgw
|   `-- ceph.keyring

Later the idea is to use ceph-disk.
So basically this is how I set an OSD (manually):

$ sudo docker run -ti -v /etc/ceph:/etc/ceph -v /var/lib/ceph/:/var/lib/ceph/ -v /dev/:/dev/ --net=host --privileged=true ceph/base /bin/bash
root@coreos:/# ceph-disk zap /dev/vdc ; ceph-disk prepare /dev/vdc ; ceph-disk activate /dev/vdc1                                                                                                                                   [33/38422]
Creating new GPT entries.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/vdc1              isize=2048   agcount=4, agsize=1304255 blks
         =                       sectsz=512   attr=2, projid32bit=0
data     =                       bsize=4096   blocks=5217019, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
The operation has completed successfully.
got monmap epoch 1
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.358696 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
2015-06-12 13:01:20.359526 7f843ec13900 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected 924fd2e5-c3ea-4d7f-b3c5-9bb81abefa07, invalid (someone else's?) journal
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.362257 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.533410 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
sh: 1: /sbin/hdparm: not found
2015-06-12 13:01:20.536231 7f843ec13900 -1 journal _check_disk_write_cache: pclose failed: (0) Success
2015-06-12 13:01:20.536667 7f843ec13900 -1 filestore(/var/lib/ceph/tmp/mnt.0_cGmZ) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2015-06-12 13:01:20.747410 7f843ec13900 -1 created object store /var/lib/ceph/tmp/mnt.0_cGmZ journal /var/lib/ceph/tmp/mnt.0_cGmZ/journal for osd.7 fsid fe49246f-9655-4034-ae32-7ff81a416e90
2015-06-12 13:01:20.748599 7f843ec13900 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.0_cGmZ/keyring: can't open /var/lib/ceph/tmp/mnt.0_cGmZ/keyring: (2) No such file or directory
2015-06-12 13:01:20.759898 7f843ec13900 -1 created new key in keyring /var/lib/ceph/tmp/mnt.0_cGmZ/keyring
added key for osd.7
root@coreos:/var/log/ceph# ceph-osd -i 7
starting osd.7 at :/0 osd_data /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
root@coreos:/var/log/ceph# ps x
  PID TTY      STAT   TIME COMMAND
    1 pts/1    Ss     0:00 /bin/sh /entrypoint.sh /bin/bash
    7 pts/1    S      0:00 /bin/bash
  226 ?        Ssl    0:00 ceph-osd -i 6
  599 pts/1    R+     0:00 ps x

This is pretty straightforward and easy to bootstrap.
I can easily prototype this.

@Ulexus what do you think?

Make ceph/demo launch using existing data

I really love the ceph/demo docker image because it provides a quick and simple way to spin up ceph for a development environment.

The only problem is that it currently treats every launch as a new cluster. I want to be able to keep my data when the docker container is terminated and then restart it at a later stage.

This is the command I am using to run the container:

sudo docker run --net=host -v /work/data/ceph/config:/etc/ceph -v /work/data/ceph/data:/var/lib/ceph -e MON_IP=192.168.33.12 -e CEPH_NETWORK=192.168.33.0/24 ceph/demo

Unfortunately, if I terminate the container and then launch it again, I get this error:

2015-07-19 06:33:20.903601 7fd39a6a9900 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0: (22) Invalid argument

I believe this is because the entrypoint.sh file treats each launch as a new cluster and will go and create the OSDs and MDS from scratch.

If would be awesome if the script has some checks in place so that it's able to "restore" if data exists rather than create a brand new cluster.

Ceph-calamari docker image

I'd be good to get calamari working inside docker.

I tried to once get it working by following all the instructions but got stuck in the ruby frameworks and couldn't get it going.

Update Docker Hub Ceph org with all the images

TODO

MDS names may not start with a numeric digit.

Just spotted this:

deprecation warning: MDS id 'mds.7cf73bf1fd74' is invalid and will be forbidden in a future version. MDS names may not start with a numeric digit.

An easy work around could be to do something like this on the MDS_NAME variable: : ${MDS_NAME:=a$(hostname -s)}

Build radosgw with civet web instead of apache

No apache required, easy to setup.
Will push something soon.

How about pushing images in docker hub

It would be better to push these images in docker hub so that we don't need to build them again 😃

Switching to ceph/daemon and/or ceph/config

@leseb - for the impatient (that's me ;) are you able to update the readme for those wishing to switch from ceph/mon, ceph/osd to ceph/daemon?

I take it that ceph/osd, ceph/mon etc will be removed at some point in the near future.
Also, I think it might be worthwhile using ceph/config, how would you go about transitioning to this from a static config?

What is the .s3cfg file for ceph container

I'm running ceph/demo in localhost and try to test the rgw service.

Then I download the s3cmd. But what's the default .s3cfg file should I use? Can anyone help to give an sample file for me to test?

Ansible to generate Ceph config/keys and distribute them

The way we currently generates the ceph.conf is really static.
Even if we use etcd or consul we still need a default raw implementation.
I believe Ansible could help on that matter.

After discussing with @SamYaple, it seems that Kolla has already a solution for this.
https://github.com/stackforge/kolla/blob/master/specs/ansible-multi.rst

ceph images in error state

https://registry.hub.docker.com/u/ceph/config/builds_history/151300/

Not sure why, but:
buh82pyfb6yjciwqaspwweb Error 2015-03-05 00:06:05 2015-03-05 00:17:02

Could someone retrigger the build and/or fix the issue ?

Thanks

Ceph OSD docker bug

Just put this here as a reminder and potential blocker for Docker OSD deployments.
http://tracker.ceph.com/issues/10763

apt repos out of sync

rebuilding mds give me this error :

E: Can't find a source to download version '2.9.2-4ubuntu4' of 'fuse:amd64'

adding apt-get update to the Dockerfile fix it.

Use data container instead of bind mount?

This was suggested by @sdake on the openstack-dev ML.
This can potentially be done but I believe this adds (unnecessary?) complexity especially moving toward etcd or consul to store cluster configs. I think we should focus on etcd.

@Ulexus what do you think?

Radosgw docs are missing something

I'm trying to run ceph/radosgw:firefly instead of my perfectly running bobrik/radosgw, but errors appear in /var/log/apache2/error.log:

root@958cfdfeba97:~# cat /var/log/apache2/error.log
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.18. Set the 'ServerName' directive globally to suppress this message
[Thu Mar 12 19:52:38.268056 2015] [:notice] [pid 83:tid 139648254072704] FastCGI: process manager initialized (pid 83)
[Thu Mar 12 19:52:38.268450 2015] [mpm_event:notice] [pid 67:tid 139648254072704] AH00489: Apache/2.4.7 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141 configured -- resuming normal operations
[Thu Mar 12 19:52:38.268487 2015] [core:notice] [pid 67:tid 139648254072704] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
[Thu Mar 12 19:53:20.650887 2015] [fastcgi:error] [pid 84:tid 139648168228608] (2)No such file or directory: [client 192.168.2.89:37187] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:53:20.650955 2015] [fastcgi:error] [pid 84:tid 139648168228608] [client 192.168.2.89:37187] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:55:41.381134 2015] [fastcgi:error] [pid 84:tid 139648159835904] (2)No such file or directory: [client 192.168.2.89:37258] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:55:41.381162 2015] [fastcgi:error] [pid 84:tid 139648159835904] [client 192.168.2.89:37258] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:56:29.117816 2015] [fastcgi:error] [pid 86:tid 139648168228608] (2)No such file or directory: [client 192.168.2.89:37291] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:56:29.117890 2015] [fastcgi:error] [pid 86:tid 139648168228608] [client 192.168.2.89:37291] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:56:37.835153 2015] [fastcgi:error] [pid 86:tid 139648159835904] (2)No such file or directory: [client 192.168.2.89:37296] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:56:37.835180 2015] [fastcgi:error] [pid 86:tid 139648159835904] [client 192.168.2.89:37296] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:56:47.834311 2015] [fastcgi:error] [pid 84:tid 139648168228608] (2)No such file or directory: [client 192.168.2.89:37301] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:56:47.834339 2015] [fastcgi:error] [pid 84:tid 139648168228608] [client 192.168.2.89:37301] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"
[Thu Mar 12 19:57:15.163202 2015] [fastcgi:error] [pid 84:tid 139648159835904] (2)No such file or directory: [client 192.168.2.89:37329] FastCGI: failed to connect to server "/var/www/s3gw.fcgi": connect() failed
[Thu Mar 12 19:57:15.163229 2015] [fastcgi:error] [pid 84:tid 139648159835904] [client 192.168.2.89:37329] FastCGI: incomplete headers (0 bytes) received from server "/var/www/s3gw.fcgi"

Apparently, something is missing in my config that should be noted in README.md:

[client.radosgw.gateway]
keyring = /etc/ceph/keyring.radosgw.gateway
rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
log file = /var/log/ceph/client.radosgw.gateway.log
rgw print continue = false
rgw_dns_name = s3.ceph.something.something

Would it be better to mount the OSD's inside the docker container rather than passing them through?

At the moment, ceph-osd relies on the physical ceph osd drives to be passed in via mounts which should appear under /var/lib/ceph/osd/ceph-$id

Note: The rest of this relates to running multiple OSD's in a single docker container.

Unfortunately, you can't pass a single /var/lib/ceph/osd as a host mount for all the mounts below it. And I'm thinking that it should be possible to hotswap osd's in and out on the fly without restarting the other osd's. Unfortunately, I don't see a way you can add volume mounts to an already running container without restarting it with the commandline changed. And passing in a top level /var/lib/ceph/osd results in that being mounted as tmpfs which breaks things.

Maybe another approach is to change the bootstrap init script to search for block devices that have the right uuid (4fbd7e29-9d25-41b8-afd0-062c0ceff05d), mount them temporarily and read the whoami file, then mount them as the correct place internal to the image. Then add other helpers to stop osd's, or add new ones on the fly which you can call through exec on the running ceph-osd.

I have a bash script that is already doing the osd id discovery and mounts on the outside which does that now and is based on what ceph-disk does.

If this makes any sense I'm prepared to fork this and give it a try, I'll post the results back here.

Does this seem sensible? or are there other reasons why you shouldn't mount block devices inside the container?

Can't run ceph/config

On a CoreOS host I run:

core@coreos ~ $ etcdctl ls
/coreos.com

docker run --net=host -e ETCDCTL_PEERS=http://192.168.0.20:4001 -e MON_IP=192.168.0.20 -e MON_NAME=etcdmon -e CLUSTER=ceph -v /etc/ceph:/etc/ceph ceph/config
Configuration found for cluster ceph. Writing to disk.
Error: 100: Key not found (/ceph-config/ceph/ceph.conf) [30]

Not sure why the script believes that /ceph-config/ceph/done exists...
Tried to dig a bit but without much luck.

@Ulexus did you get a chance to test that new container?

Unable to restart ceph cluster from scratch

Using ceph-config, the script fails on "Attempting to pull monitor map from existing cluster." because if the cluster is down, then there is no way to retreive the map from an existing monitor, and the script stops because of -e option, leaving the configuration in a locked state.

Add detailed instruction for demo docker container

I would like to try demo docker container but I'm confused about the parameters.

Here's my error message and I have no idea how to fix it. More instruction is really welsome 😃

➜  demo git:(master) docker run -d --net=host -v /etc/ceph:/etc/ceph -e MON_IP=172.17.42.1 -e CEPH_NETWORK=172.17.0.0/24 ceph/demo                                                                                  
ee3fa8858386595b7e66d7eeebf016694b0ee59a5a19a0d4b2a7f10d3fce5e87
➜  demo git:(master) docker logs -f ee3fa8858386595b7e66d7eeebf016694b0ee59a5a19a0d4b2a7f10d3fce5e87                              
creating /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /tmp/ceph.mon.keyring
importing contents of /etc/ceph/ceph.mon.keyring into /tmp/ceph.mon.keyring
2015-05-20 10:07:18.471878 7f54afc4b8c0 -1 unable to find any IP address in networks: 96.126.0.0/24

Contact details

Hey @Ulexus
Just to let you know, I'm on freenode and oftc as leseb. So feel free to ping me if needed.
I'm based in Paris. UTC +1
Are you on IRC?

Thanks!

Usage of confd to make the repo more universal

Proposed by @hunter in #28 .
Discussion can continue here.

hdparm missing

I see some errors related to hdparm missing from ceph-osd. It's not installed in the image.

Ceph AIO container

I know running multiple processes defeats Docker's philosophy but I believe some use cases might require a simple standalone container running all the Ceph processes.
Actually it's easier to bootstrap.

@Ulexus thoughts?

KV store problems

I'm just starting to play with the KV setup, and I am finding it's pretty rough. I'm adding this issue as much for my own notes as for anything else. Also, these are just my opinions, so feel free to provide feedback/arguments.

Problems (update the following as necessary or discovered):

• The default schema for the ceph.conf construction doesn't get populated in a usable way
• The schema for ceph.conf isn't documented
• Some segments in the KV schema are directory-separated; others are _-separated
• Some schema changes I would prefer:
  • /auth/fsid should be /fsid (IMO, this is is only tertiarily auth: it's the cluster id)
  • Keyrings should be in /keyrings; examples: /keyrings/mon, /keyrings/bootstrapOsd (maybe even /keyrings/bootstrap/osd
  • /auth/cephx_* should be /auth/cephx/* (though this seems redundant with the /globals/ section)
• If mandatory directories in the KV store don't exist, the config construction dies -- need valid defaults
• Public and Cluster networks should be stored in the KV
• It doesn't appear the bootstrap keyrings actually get added to the auth database
• I also really like the idea of merging a user-supplied, whole ceph.conf with a confd-generated one.

NOTE: Many of these would be breaking changes, so I would also like to get an idea of how many people are actually using the KV backend so that we can see if we need to add backward compatibility shims.

ceph/config always regenerating config

etcdctl get --consistent ${CLUSTER_PATH}/done 2>/dev/null >/dev/null
if [ $? -ne 0 ]; then
  echo "Configuration found for cluster ${CLUSTER}. Writing to disk."

It probably should check if it's -eq 0.

Kubernetes example v1 api is using wrong json

According to V1 api volume section should use "hostpath" key instead of "source"->"hostdir".

http://kubernetes.io/third_party/swagger-ui/#!/api%2Fv1/createReplicationController

Import https://github.com/bobrik/docker-radosgw as radosgw

Because I didn't ever actually get around to writing a radosgw, I was looking around at what's out there. @bobrik's image looks good to me, but he uses nginx (which I prefer, but is not reflective of what is on ceph.com). I asked him, and he's happy to supply his code.

@leseb Does that work for you, or do you want to try and keep with an Apache installation, consistent with the docs on ceph.com?

Bootstrap cluster not caterred for providing your own ceph.conf

As the pre-generated ceph.conf is very basic when bootstrapping a new cluster, it's often better to supply one.

However, the entrypoint.sh scripts on the mon and daemon skip the generation of admin & monitor keys if ceph.conf exists. It'd be better to prompt for them to be created, or just create them if ceph.conf is there, but those files are missing.

So in my case I had to run the authtool directly to complete the missing steps.

user_xattr

the demo container requires user_xattr

I was going to update the readme with a remount example but I'm not sure the best way to do it

something like this:

 sudo mount -o remount,user_xattr,rw /var

RFC: rename ceph/ceph-base to just ceph/base

Now that we have the Ceph namespace on Docker, it seems to me that we should collapse the name for base (and all the other components).

Add necessary things for rgw in base or build a new one for rgw only?

I don't really the idea of having apache running on MON/OSD/MDS.
But having 2 base (even if one is named for rgw) is not really sophisticated.

I'm more in favour of new image for RGW base.

@Ulexus what do you think?

rbd-volume: can't mount to host filesystem/link from other containers

I have a container with mounted RBD image as /mnt/rbd volume. Inside this container, I can see the contents of /mnt/rbd. But when I try to mount it to the host filesystem mountpoint (-v /host/path/mountpoint:/mnt/rbd) or use link to that volume from other containers with the --volumes-from Docker run option, I see only an empty directory.
Correct me if I'm wrong, but I think it's Dockers issue that cannot pass the mount point inside the container through the -v or --volumes-from options?

ceph-disk notes

I'm really wanting that fully automated life cycle goal that the non-docker version as far as I can tell have.

So, toward that goal I just wanted somewhere to capture some ceph-disk notes and get peoples comments. Sorry if this is a bit long winded. I wish github had a discussion type of system. Anyway, onwards...

Right now we're able to use, ceph-disk prepare as a step toward getting a ceph osd up.
I use it, because I run alongside my OSD spinning disks, 2 SSD's for journalling.

I prepare my OSD's like this:

docker run --rm --net=host --privileged=true -v /etc/ceph:/etc/ceph -v /dev:/dev ceph/base ceph-disk prepare --cluster ceph --cluster-uuid 683f638b-e2d1-40c2-969f-907d961a7d24 --fs-type xfs /dev/sda /dev/sde

Assuming my cluster-uuid is that above (gotten from my ceph.conf) and /dev/sda is my main data disk, /dev/sde is my journal disk.

Very conveniently, if I run the command again with a different data disk, and then use the same journal disk, ceph-disk will create another journal partition. In actual fact, it creates a symbolic link to the journal disk on the main osd data disk.

Where everything falls over, is with ceph-disk activate. It's just not going to work actually. The reason it's not going to work is because it expects:

1. to be running natively (not docker).
2. it auto-detects what OS you're using and what init system it is. It's most likely going to get that totally wrong. And besides that, the init system used in our images is not on the list.
3. ceph-disk activate also writes the OSD ID to the whoami file so you could move the disk elsewhere in future.

So, either ceph-disk would need to be changed to be docker aware (which would likely be very tied to this implementation), or we override it. Or jump through hoops. Or use systemd inside the image... not sure I like that idea but I've heard it's possible.

ceph-disk code is here. It's really just a python script that uses sgdisk, blkid and a few other tools to get the job done.
https://github.com/ceph/ceph/blob/master/src/ceph-disk

I think the override idea is the easiest.
As ceph-disk activate doesn't work, what I do is currently:
Assuming my OSD is /dev/sda & is osd 10

sudo mkdir -p /osd/tmp
sudo mount /dev/sda1 /osd/tmp
sudo echo 10 | tee /osd/tmp/whoami
sudo umount /osd/tmp

Which just writes the whoami file. And combine that with my mount-osds script.
If the disks ever need to be moved to another machine, then knowing in advance what disk is becomes a bit tedious. This is where I think my mount-osds is really useful.. It'll mount them to /osd/ceph-$id. It's not really entirely cluster aware but could be made to be. i.e. currently semi-hardcoded to a cluster named ceph. It's actually doing a bit of what ceph-disk does but does it in shell script so it can be run natively from CoreOS without requiring python and can run outside a container.

What I do is mount the disks by calling mount-osds before the docker ceph-osd container is started.
Or just calling it as an ExecStartPre= from a systemd service file or similar.

What I am thinking is merging some of this functionality with my ceph-tools scripts. Override ceph-disk activate to actually do the above and the bits from the bootstrap-osd, then write a startup file to /etc/service/ceph-osd/run and start it with sv start.

I'm not sure if this makes a terrible lot of sense to people yet... later I'll whip up an implementation and you can have a peek. But I'd like to see @leseb daemon changes merged to make it work off that. Anyway, those are my thoughts currently.

Would love to know how to add into systemd to call some custom udev rules, e.g. like if a ceph osd disk were moved it could run through a setup or restart process automagically. Has anyone actually done that here?