GithubHelp home page GithubHelp logo

ceph-cookbook's Introduction

Ceph - a scalable distributed storage system

See https://ceph.com/ for current information about Ceph.

Status

Issue Backporting

Contributing Code

Most of Ceph is dual-licensed under the LGPL version 2.1 or 3.0. Some miscellaneous code is either public domain or licensed under a BSD-style license.

The Ceph documentation is licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

Some headers included in the ceph/ceph repository are licensed under the GPL. See the file COPYING for a full inventory of licenses by file.

All code contributions must include a valid "Signed-off-by" line. See the file SubmittingPatches.rst for details on this and instructions on how to generate and submit patches.

Assignment of copyright is not required to contribute code. Code is contributed under the terms of the applicable license.

Checking out the source

Clone the ceph/ceph repository from github by running the following command on a system that has git installed:

git clone [email protected]:ceph/ceph

Alternatively, if you are not a github user, you should run the following command on a system that has git installed:

git clone https://github.com/ceph/ceph.git

When the ceph/ceph repository has been cloned to your system, run the following commands to move into the cloned ceph/ceph repository and to check out the git submodules associated with it:

cd ceph
git submodule update --init --recursive --progress

Build Prerequisites

section last updated 27 Jul 2023

Make sure that curl is installed. The Debian and Ubuntu apt command is provided here, but if you use a system with a different package manager, then you must use whatever command is the proper counterpart of this one:

apt install curl

Install Debian or RPM package dependencies by running the following command:

./install-deps.sh

Install the python3-routes package:

apt install python3-routes

Building Ceph

These instructions are meant for developers who are compiling the code for development and testing. To build binaries that are suitable for installation we recommend that you build .deb or .rpm packages, or refer to ceph.spec.in or debian/rules to see which configuration options are specified for production builds.

To build Ceph, make sure that you are in the top-level ceph directory that contains do_cmake.sh and CONTRIBUTING.rst and run the following commands:

./do_cmake.sh
cd build
ninja

do_cmake.sh by default creates a "debug build" of Ceph, which can be up to five times slower than a non-debug build. Pass -DCMAKE_BUILD_TYPE=RelWithDebInfo to do_cmake.sh to create a non-debug build.

Ninja is the buildsystem used by the Ceph project to build test builds. The number of jobs used by ninja is derived from the number of CPU cores of the building host if unspecified. Use the -j option to limit the job number if the build jobs are running out of memory. If you attempt to run ninja and receive a message that reads g++: fatal error: Killed signal terminated program cc1plus, then you have run out of memory. Using the -j option with an argument appropriate to the hardware on which the ninja command is run is expected to result in a successful build. For example, to limit the job number to 3, run the command ninja -j 3. On average, each ninja job run in parallel needs approximately 2.5 GiB of RAM.

This documentation assumes that your build directory is a subdirectory of the ceph.git checkout. If the build directory is located elsewhere, point CEPH_GIT_DIR to the correct path of the checkout. Additional CMake args can be specified by setting ARGS before invoking do_cmake.sh. See cmake options for more details. For example:

ARGS="-DCMAKE_C_COMPILER=gcc-7" ./do_cmake.sh

To build only certain targets, run a command of the following form:

ninja [target name]

To install:

ninja install

CMake Options

The -D flag can be used with cmake to speed up the process of building Ceph and to customize the build.

Building without RADOS Gateway

The RADOS Gateway is built by default. To build Ceph without the RADOS Gateway, run a command of the following form:

cmake -DWITH_RADOSGW=OFF [path to top-level ceph directory]

Building with debugging and arbitrary dependency locations

Run a command of the following form to build Ceph with debugging and alternate locations for some external dependencies:

cmake -DCMAKE_INSTALL_PREFIX=/opt/ceph -DCMAKE_C_FLAGS="-Og -g3 -gdwarf-4" \
..

Ceph has several bundled dependencies such as Boost, RocksDB and Arrow. By default, cmake builds these bundled dependencies from source instead of using libraries that are already installed on the system. You can opt to use these system libraries, as long as they meet Ceph's version requirements. To use system libraries, use cmake options like WITH_SYSTEM_BOOST, as in the following example:

cmake -DWITH_SYSTEM_BOOST=ON [...]

To view an exhaustive list of -D options, invoke cmake -LH:

cmake -LH

Preserving diagnostic colors

If you pipe ninja to less and would like to preserve the diagnostic colors in the output in order to make errors and warnings more legible, run the following command:

cmake -DDIAGNOSTICS_COLOR=always ...

The above command works only with supported compilers.

The diagnostic colors will be visible when the following command is run:

ninja | less -R

Other available values for DIAGNOSTICS_COLOR are auto (default) and never.

Building a source tarball

To build a complete source tarball with everything needed to build from source and/or build a (deb or rpm) package, run

./make-dist

This will create a tarball like ceph-$version.tar.bz2 from git. (Ensure that any changes you want to include in your working directory are committed to git.)

Running a test cluster

From the ceph/ directory, run the following commands to launch a test Ceph cluster:

cd build
ninja vstart        # builds just enough to run vstart
../src/vstart.sh --debug --new -x --localhost --bluestore
./bin/ceph -s

Most Ceph commands are available in the bin/ directory. For example:

./bin/rbd create foo --size 1000
./bin/rados -p foo bench 30 write

To shut down the test cluster, run the following command from the build/ directory:

../src/stop.sh

Use the sysvinit script to start or stop individual daemons:

./bin/init-ceph restart osd.0
./bin/init-ceph stop

Running unit tests

To build and run all tests (in parallel using all processors), use ctest:

cd build
ninja
ctest -j$(nproc)

(Note: Many targets built from src/test are not run using ctest. Targets starting with "unittest" are run in ninja check and thus can be run with ctest. Targets starting with "ceph_test" can not, and should be run by hand.)

When failures occur, look in build/Testing/Temporary for logs.

To build and run all tests and their dependencies without other unnecessary targets in Ceph:

cd build
ninja check -j$(nproc)

To run an individual test manually, run ctest with -R (regex matching):

ctest -R [regex matching test name(s)]

(Note: ctest does not build the test it's running or the dependencies needed to run it)

To run an individual test manually and see all the tests output, run ctest with the -V (verbose) flag:

ctest -V -R [regex matching test name(s)]

To run tests manually and run the jobs in parallel, run ctest with the -j flag:

ctest -j [number of jobs]

There are many other flags you can give ctest for better control over manual test execution. To view these options run:

man ctest

Building the Documentation

Prerequisites

The list of package dependencies for building the documentation can be found in doc_deps.deb.txt:

sudo apt-get install `cat doc_deps.deb.txt`

Building the Documentation

To build the documentation, ensure that you are in the top-level /ceph directory, and execute the build script. For example:

admin/build-doc

Reporting Issues

To report an issue and view existing issues, please visit https://tracker.ceph.com/projects/ceph.

ceph-cookbook's People

Contributors

alram avatar bstaz avatar cheekyhalf avatar dmlb2000 avatar git-harry avatar gregsfortytwo avatar guilhem avatar hufman avatar hunter avatar inktankps avatar johnae avatar jperville avatar jsuchome avatar klamontagne avatar kri5 avatar laradji avatar liewegas avatar mancdaz avatar mattt416 avatar mdsteveb avatar mgalkiewicz avatar mickmccarthy avatar mmgaggle avatar odyssey4me avatar pentabular avatar scarvalhojr avatar sorccu avatar tas50 avatar tv42 avatar yuryu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

ceph-cookbook's Issues

All In One recipe

Be able to create a "all-in-one" box to be able to have a cluster tested by TK

rhel rpm-release files don't exist on ceph.com

Whilst not technically a cookbook issue, it's something we can fix in the cookbook if we want to.

When using rhel platform family, the ceph repo is setup by installing the ceph-release file for the relevant release. Eg, for dumpling, the following file is supposed to be installed:

http://ceph.com/rpm-dumpling/el6/x86_64/ceph-release-1-0.el6.noarch.rpm

This is built from the attribute:

default['ceph']['rhel']['stable']['repository'] = "#{node['ceph']['repo_url']}/rpm-#{node['ceph']['version']}/el6/x86_64/ceph-release-1-0.el6.noarch.rpm"

Unfortunately, the ceph-release file does not exist. if this in itself is a problem, then I can report that bug elsewhere (where?). If it's to remain that way, then we can fix it in the cookbooks by manually creating the ceph.repo file using the yum_repository provider (from the yum cookbook)

Set max_open_files for large clusters

The ansible cookbook defaults to setting max_open_files to 128k. Users of the Chef cookbook can say node['ceph']['config']['global']['max open files'] = 131072, but perhaps this should be the default?
It seems like the setting defaults to whatever ulimit is set to.

ceph_cephfs resource tries to mount again (mons order changed)

I currently have the following mounted fs in my fstab:

1.2.3.1:6789,1.2.3.3:6789,1.2.3.4:6789,1.2.3.2:6789:/ /media/ceph ceph _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 0 0

chef-client tries to mount the "new" fs and mount exits unsuccessfuly:

[2014-12-03T14:50:19-05:00] ERROR: mount[mount /media/ceph] (/var/chef/cache/cookbooks/ceph/providers/cephfs.rb line 35) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '16'
---- Begin output of mount -t ceph -o _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 1.2.3.1:6789,1.2.3.4:6789,1.2.3.3:6789,1.2.3.2:6789:/ /media/ceph ----
STDOUT: mount error 16 = Device or resource busy
STDERR:
---- End output of mount -t ceph -o _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 1.2.3.1:6789,1.2.3.4:6789,1.2.3.3:6789,1.2.3.2:6789:/ /media/ceph ----
Ran mount -t ceph -o _netdev,name=cephfs.mymachine,secretfile=/etc/ceph/ceph.client.cephfs.mymachine.secret 1.2.3.1:6789,1.2.3.4:6789,1.2.3.3:6789,1.2.3.2:6789:/ /media/ceph returned 16
[2014-12-03T14:50:19-05:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

It seems that the order of mons is not kept between runs. (1,3,4,2 vs 1,4,3,2)

How to set osd location information?

I took a quick browse around and didn't see a way to accomplish this:

I want to set some values in ceph.conf on a per-node basis so that I can set OSD location information. For me, I'm trying to set a datacenter name so that OSDs will be placed in the correct buckets in the crush map. I'm using this as a reference for what I need to do with Ceph: http://dachary.org/?p=2536

So, it looks like for each node, I need to add osd_crush_update_on_start = 1 and osd_crush_location = dcname to ceph.conf. I tried dropping these values into the node's JSON (in the normal->ceph section) via knife node edit, but no such luck.

Is there a way to do this or is a patch needed?

Add systemd support

The cookbook currently supports sysvinit and upstart. Newer systems are using systemd, and some of the service restart logic needs to be modified to support it.
This is blocking #145

radosgw should support civetweb frontend

Ceph Firefly supports a standalone civetweb frontend for radosgw, instead of needing a separate server like Apache. Ceph Giant is slated to make it even more supported. The cookbook should support using it as well as Apache.

ceph_cephfs mount options

When using provider ceph_cephfs to mount a cephfs volume, there is no way to specify mount options.
Wouldn't it be wise to enable some sort of options, for instance adding "noatime"?

Kernel module not installed

Should the rbd kernel module be installed with the cookbook? After doing an install I'm finding that 'modprobe rbd' can't find the module.

deploy fails when public_network contains more than one network

I'm new to ruby, chef and the cookbook so it's possible my analysis is flawed, and I'm unable to suggest a patch, but please bear with me.

I'm having trouble deploying a configuration which lists multiple public networks. The problem seems to be find_node_ip_in_network which does not check for this possibility and instead passes the full string directly to IPAddr.new, which fails.

To be more specific, in my environment I set a default attribute:

"public network": "192.168.10.0/24,192.168.20.0/24"

e.g. listing two subnets. Ceph supports this as described here: http://ceph.com/docs/master/rados/configuration/network-config-ref/#network-config-settings

In any case it doesn't look like either find_node_ip_in_network or its caller mon_addresses make any any effort to handle multiple networks. The result is that when chef-client runs, it dies with a backtrace:

ArgumentError: invalid address
/opt/chef/embedded/lib/ruby/1.9.1/ipaddr.rb:544:in `in6_addr'
/opt/chef/embedded/lib/ruby/1.9.1/ipaddr.rb:481:in `initialize'
/opt/chef/embedded/lib/ruby/1.9.1/ipaddr.rb:401:in `new'
/opt/chef/embedded/lib/ruby/1.9.1/ipaddr.rb:401:in `mask!'
/opt/chef/embedded/lib/ruby/1.9.1/ipaddr.rb:488:in `initialize'
/var/chef/cache/cookbooks/ceph/libraries/default.rb:42:in `new'
/var/chef/cache/cookbooks/ceph/libraries/default.rb:42:in `find_node_ip_in_network'
/var/chef/cache/cookbooks/ceph/libraries/default.rb:87:in `block in mon_addresses'
/var/chef/cache/cookbooks/ceph/libraries/default.rb:87:in `map'
/var/chef/cache/cookbooks/ceph/libraries/default.rb:87:in `mon_addresses'
/var/chef/cache/cookbooks/ceph/recipes/mon.rb:108:in `from_file'
....

test vdis fail to mount

I've been trying to test this cookbook and kitchen test fails to create any of the vms.

default: SSH auth method: private key
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...
default: Warning: Connection timeout. Retrying...

I enabled the VirtualBox GUI and the box reported 'Could not read from the boot medium! System halted.'

I personally think this has to do with this section of the vagrantfile used for testing.

Sometimes I get this error:

#<Vagrant::Errors::VBoxManageError: There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["createhd", "--filename", "disk-0", "--size", "1000"]

Stderr: 0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Failed to create medium
VBoxManage: error: Could not create the medium storage unit '/Users/doctordoom/workspace/work/ceph-cookbook/.kitchen/kitchen-vagrant/kitchen-ceph-cookbook-osd-ubuntu-1204/disk-0.vdi'.
VBoxManage: error: VDI: cannot create image '/Users/doctordoom/workspace/work/ceph-cookbook/.kitchen/kitchen-vagrant/kitchen-ceph-cookbook-osd-ubuntu-1204/disk-0.vdi' (VERR_ALREADY_EXISTS)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component MediumWrap, interface IMedium
VBoxManage: error: Context: "RTEXITCODE handleCreateMedium(HandlerArg*)" at line 448 of file VBoxManageDisk.cpp
>

Please fix this customization and try again.

I am running VirtualBox 5.0.0.

add_bootstrap_hint fails within within virtualbox

This is the last line I get before the Chef run within a vagrant issue hangs:

- execute ceph --admin-daemon '/var/run/ceph/ceph-mon.default-ubuntu-1404.asok' add_bootstrap_peer_hint 10.0.2.15:6789

My testing situation is somewhat different from this cookbook. I am pulling this cookbook in solely for the ceph_client resource so I can set up all the bits needed to mount an RBD volume using Chef and I can't get it working within my test environment.

Ubuntu 14.04 installs the wrong version of Ceph

Ubuntu 14.04's repos have version 0.79 and version 0.80 available in the repos, and so while the Chef cookbook says to install one version, another version entirely gets installed. This is mainly a problem when trying to install a specific version on a new mon node to join an existing cluster with an older version running, because the new mon isn't able to connect.

Cephfs doesn't support switching from kcephfs to fuse

Currently, if kcephfs is mounted and the recipe tries to mount fuse, it will fail because the mountpoint directory is not empty.
It should notice that kcephfs is mounted and try to unmount it first.
Probably will need another test recipe to mount it in one mode and then remount in the other mode.

Chef does not generate per mon/mds/osd sections

In ceph.conf chef should generate per service sections in order to start them from init.d script.

For example my osd will not start unless there is an entry in ceph.conf:

...
[osd.0]
  host = n1cc
...

recipe[ceph::apt] breaks with apt cookbook >= 2.7.0

I just found out that the recipe[ceph::apt] is broken with recent (>= 2.7.0) version of the apt cookbook.

==> backends: Chef::Exceptions::ValidationFailed
==> backends: ----------------------------------
==> backends: Option package_name's value * does not match regular expression [/^([a-z]|[A-Z]|[0-9]|_|-|\.)+$/]
==> backends: 
==> backends: 
==> backends: Cookbook Trace:
==> backends: ---------------
==> backends:   /tmp/vagrant-chef/a18f1b1b56cdb4f3151fc70d8ad2a838/cookbooks/ceph/recipes/apt.rb:13:in `block in from_file'
==> backends:   /tmp/vagrant-chef/a18f1b1b56cdb4f3151fc70d8ad2a838/cookbooks/ceph/recipes/apt.rb:12:in `from_file'
==> backends: 
==> backends:   /tmp/vagrant-chef/a18f1b1b56cdb4f3151fc70d8ad2a838/cookbooks/ceph/recipes/repo.rb:3:in `from_file'
==> backends: 
==> backends:   /tmp/vagrant-chef/a18f1b1b56cdb4f3151fc70d8ad2a838/cookbooks/pm-ceph/recipes/default.rb:29:in `from_file'
==> backends:   /tmp/vagrant-chef/a18f1b1b56cdb4f3151fc70d8ad2a838/cookbooks/pm-ceph/recipes/mon.rb:10:in `from_file'
==> backends: 

As shown by the above trace, the culprit is https://github.com/ceph/ceph-cookbook/blob/master/recipes/apt.rb#L13

apt_preference 'ceph_repo' do
  package_name '*' # <--- here
  pin 'origin "ceph.com"'
  pin_priority '1001'
end

Which triggers sous-chefs/apt@d9da784#diff-e34be685fe9b4a095faaba48630cb896

According to the comments of the above commit, it seems that '*' is not a legal for the apt_preference package_name, and that it will remain illegal in the future.

For now, my workaround is to lock apt version to the last pre-2.7.0 version, 2.6.1.

How to handle osd disk failure?

Hi All!

I was thinking about how the coobook handles disk failures. What would be an operational procedure in case of an OSD disk dying with this cookbook? Has anyone have experience about this? Thanks in advance!

have these ever been tested with centos??

This cookbook seems to be horribly broken when running on centos. It tries to add an rpm as a repo, doesn't add repoforge but requires packages for it, doesn't include the gpg keys for the stable and testing repos (which need one), and in recipes/mon.rb, is_mon isn't before ceph::default so default never picks up that it's actually a monitor host. I feel like I'm QA for this. Please, please test your code in cent. or tell me if I'm doing something horribly wrong here.

Down OSDs are (inadvertantly) brought up by chef-client

This behavior has caused problems for me in a few ways.

We have chef-client running every 30 minutes (daemonized). Every time it runs, it brings up all OSDs, even ones that have been marked down due to starting to fail (resulting in flapping up/down), or manually downed for whatever reason. This is pretty anti-social. (This morning I had to take a couple nodes' worth of OSDs down for a network outage, but didn't want to take the entire systems down so that I could verify network had been restored properly when it was finished; only to find that they'd already come back up on their own which was definitely not what I wanted or intended.)

Are there any parameters I can tweak to prevent this? How are others dealing with this? I know I can probably kill chef-client on the node for the duration but this isn't exactly optimal either.

fedora-20 doesn't start OSDs

When trying to add fedora-20 to the test kitchen, besides failing #135, it doesn't install the udev hooks to run ceph-disk-activate automatically.
This may be because it is installing ceph-0.81.0 from the Fedora repos instead of the specified version from the Ceph repos.

Centos6.5 fails to ceph-disk-activate

The ceph-disk-prepare script seems to make the journal point to /dev/disk/by-partuuid/{uuid}, but Centos6.5 doesn't have a /dev/disk/by-partuuid directory.

Ceph Giant doesn't seem to properly start mds

When setting the version attribute to giant and running the test, it freezes when trying to mount cephfs. The ceph-mds daemon is running, but there aren't any mds lines from ceph -s.

kitchen test aio-centos-65 fails due to missing /lib/lsb/init-functions

The recipe tries to start the ceph service, but that fails because the init script relies on the standard /lib/lsb/init-functions library, which is provided by the redhat-lsb package.
Should we ensure that package is installed in recipes::rpm, or something similar? It happens with both dumpling and emperor.

Automated OSD HDD setup

There should be an optional attribute that allows you to specify that you want the osd's prepared automatically.

It should then allow you to specify a regex for the drives that should be automatically allocated (wholly) for OSD's. eg: [hvs]d[^a]

Based on this regex, the node's attributes should automatically be populated with the required structure as per the recipe:

"ceph": {
  "osd_devices": [
    { "device": "/dev/sdb" },
    { "device": "/dev/sdc" },
    { "device": "/dev/sdd" },
    { "device": "/dev/sde" },
    { "device": "/dev/sdf" }
  ]
}

This will be a good starting point - later it can be extended to allow automation for the way journaling is done and special attributes like dmcrypt.

Can't mount ceph-fuse on centos6.5

The cookbook fails to mount ceph-fuse volumes, because /sbin/mount.fuse.ceph is missing. The package installed /usr/sbin/mount.fuse.ceph, but apparently didn't copy it to /sbin?
Centos6.5 shows this, but Centos7 installs the file in the correct place. I tested both Dumpling and Firefly.

ERROR: execute[peer ["lo", "eth0", "bond0", "br0"]] (ceph::mon line 87) had an error: Expected process to exit with [0], but received '255'"

I've received the following error in my chef client.log

ERROR: execute[peer ["lo", "eth0", "bond0", "br0"]](ceph::mon line 87) had an error: Expected process to exit with [0], but received '255'"

After investigating the /etc/ceph/ceph.conf
the mon line has

mon host = lo, eth0, bond0, br0, eth0, lo, br0, bond0

Instead of the IP addresses of the hosts. i'm not sure why this has happened. I followed the documentation at http://wiki.ceph.com/02Guides/Deploying_Ceph_with_Chef

Thanks,

ceph should reuse fsid across cluster

If the fsid isn't set in the attributes, it will make up a new one. It should instead do a search across the cluster to find it, the mon nodes for example.

ceph-mon requires mon-data path

Trying to bootstrap a cluster here.
I generated fsid using uuidgen and set it as an environment attribute, and did the same for the mon_initial_members.
The following chef-client run fails with this:

[2013-05-08T16:24:34+00:00] INFO: Processing execute[ceph-mon mkfs] action run (ceph::mon line 49)
--conf/-c        Read configuration from the given configuration file
-d               Run in foreground, log to stderr.
-f               Run in foreground, log to usual location.
--id/-i          set ID portion of my name
--name/-n        set name (TYPE.ID)
--version        show version and quit

   --debug_ms N
        set message debug level (e.g. 1)

================================================================================
Error executing action `run` on resource 'execute[ceph-mon mkfs]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring' ----
STDOUT: --conf/-c        Read configuration from the given configuration file
-d               Run in foreground, log to stderr.
-f               Run in foreground, log to usual location.
--id/-i          set ID portion of my name
--name/-n        set name (TYPE.ID)
--version        show version and quit

   --debug_ms N
        set message debug level (e.g. 1)
STDERR: must specify '--mon-data=foo' data path
usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
  --debug_mon n
        debug monitor level (e.g. 10)
  --mkfs
        build fresh monitor fs
---- End output of ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring' ----
Ran ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring' returned 1

Resource Declaration:
---------------------
# In /var/chef/cache/cookbooks/ceph/recipes/mon.rb

 49:   execute 'ceph-mon mkfs' do
 50:     command "ceph-mon --mkfs -i #{node['hostname']} --keyring '#{keyring}'"
 51:   end
 52: 

Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/ceph/recipes/mon.rb:49:in `from_file'

execute("ceph-mon mkfs") do
  action "run"
  retries 0
  retry_delay 2
  command "ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring'"
  backup 5
  returns 0
  cookbook_name "ceph"
  recipe_name "mon"
end

[2013-05-08T16:24:34+00:00] ERROR: Running exception handlers
[2013-05-08T16:24:34+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[2013-05-08T16:24:34+00:00] ERROR: Exception handlers complete
[2013-05-08T16:24:34+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-05-08T16:24:34+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: execute[ceph-mon mkfs] (ceph::mon line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring' ----
STDOUT: --conf/-c        Read configuration from the given configuration file
-d               Run in foreground, log to stderr.
-f               Run in foreground, log to usual location.
--id/-i          set ID portion of my name
--name/-n        set name (TYPE.ID)
--version        show version and quit

   --debug_ms N
        set message debug level (e.g. 1)
STDERR: must specify '--mon-data=foo' data path
usage: ceph-mon -i monid [--mon-data=pathtodata] [flags]
  --debug_mon n
        debug monitor level (e.g. 10)
  --mkfs
        build fresh monitor fs
---- End output of ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring' ----
Ran ceph-mon --mkfs -i s7r2 --keyring '/var/chef/cache/ceph-s7r2.mon.keyring' returned 1

Create Keyring Failed

What could be the problem of this, when it should create a keyring it hangs completely.

"rank": -1,
"state": "probing",
"election_epoch": 0,
"quorum": [],
"outside_quorum": [],
"monmap": { "epoch": 0,
"fsid": "6e637f1b-dfb7-4043-9371-61836cc6ff59",
"modified": "0.000000",
"created": "0.000000",
"mons": [
{ "rank": 0,
"name": "["node02"]",
"addr": "0.0.0.0:0/1"}]}} <----------------------------- This is not okay, and how can i solve this?

Improve support for Chef-Solo

Some organizations may want to deploy some of their Ceph stack using Chef-Solo, and some pieces naturally lend themselves to it, like the radosgw or iscsi nodes. This issue is a way to brainstorm for ways to improve support for Chef-Solo.

/var/run/ceph-radosgw/ disappears on reboot

0845115 introduced the /var/run/ceph-radosgw directory. Unfortunately that's not created in any of the init scripts, and disappears on reboot (Ubuntu 14.04 and likely others).

This means that on a reboot, the radosgw daemon won't work until the next chef run occurs, and even then not until radosgw is manually restarted.

trying to bootstrap a test cluster, osds are not created (auth error)

I'm trying to rebuild my test cluster from scratch with a much newer version of the cookbook than I used last time. This is on Ubuntu 14.04, installing ceph hammer.

It's building the monitor okay and it looks like the osd prepare step worked (they're all partitioned as expected anyway), but it's failing to activate them with a librados permission error.

It looks like the bootstrap-osd key is deployed to the osd hosts' /var/lib/ceph/bootstrap-osd/ceph.keyring okay but is not installed in the cluster:

# ceph auth list
installed auth entries:

client.admin
    key: AQC...XQ==
    caps: [mds] allow
    caps: [mon] allow *
    caps: [osd] allow *
client.bootstrap-mds
    key: AQ...nw==
    caps: [mon] allow profile bootstrap-mds
client.bootstrap-rgw
    key: AQ...xQ==
    caps: [mon] allow profile bootstrap-rgw
# 

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.