ceph / cn-core

Bootstrap Ceph AIO - source of cn project

/!\ This project is not maintained anymore. It was an experiment that never made it into ceph-nano. /!\

cn-core

cn-core replaces the demo.sh scenario of the ceph-container project for a faster bootstrap time, better test coverage and error handling.

cn-core init

See it in action:

[root@37c246814969 tmp]# ./cn-core init
2019/01/14 10:57:29 init: no daemon was selected. Deploying mon, mgr, osd, rgw, dash.
2019/01/14 10:57:29 init mon: run prerequisites
2019/01/14 10:57:29 init mon: writing monitor initial keyring
2019/01/14 10:57:29 init mon: writing ceph configuration file
2019/01/14 10:57:29 init mon: generating monitor map
2019/01/14 10:57:29 init mon: populating monitor store
2019/01/14 10:57:29 init mon: running monitor
2019/01/14 10:57:29 init mgr: run prerequisites
2019/01/14 10:57:29 init mgr: fetching admin keyring
2019/01/14 10:57:29 init mgr: generating manager keyring
2019/01/14 10:57:29 init mgr: running manager
2019/01/14 10:57:29 init osd: run prerequisites
2019/01/14 10:57:29 init osd: generating osd keyring
2019/01/14 10:57:30 init osd: populating osd store
2019/01/14 10:57:31 init osd: running osd
2019/01/14 10:57:33 init rgw: run prerequisites
2019/01/14 10:57:33 init rgw: generating rgw keyring
2019/01/14 10:57:33 init rgw: running rgw on port 8000
2019/01/14 10:57:33 init rgw: creating rgw user
2019/01/14 10:57:41 init dashboard: run prerequisites
2019/01/14 10:57:41 init dashboard: running dashboard on port 5000
2019/01/14 10:57:41 init ceph: running ceph health watcher

Or for daemon in particular cn-core init --daemon mon.

Full CLI example:

$ cn-core init -h
Init a Ceph cluster

Usage:
  cn-core init [flags]

Examples:
cn-core init
cn-core init --daemon mon


Flags:
  -d, --daemon string            Specify which daemon to bootstrap. Valid choices are: mon, mgr, osd, rgw, dash, health.
      --rgw-port string          Specify binding port for Rados Gateway. (default "8000")
      --dash-port string         Specify binding port for Sree dashboard. (default "5000")
      --dash-exposed-ip string   Specify binding port for Sree dashboard.
  -h, --help

cn-core's Issues

Use nautilus stable image and start octopus build

Dashboard connection issue

When I recently tried, the dashboard wasn't able to load correctly the buckets etc, I believe there is an issue in https://github.com/ceph/cn-core/blob/master/cmd/utils.go#L173

serve the dashboard from rgw itself

Good reading source https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c#rgw-zonegroup-configuration

Use tini to mimic docker_exec?

We need a similar behavior than https://github.com/ceph/ceph-container/blob/master/src/daemon/docker_exec.sh. If a process fails or partially some daemon fail we need to be able to keep the container alive.

use msgr2 mon ip style

We need to bind the mon port on 3300 and enable msgr2 in the mon_host section.

ceph-volume support

Convert this:

function bootstrap_osd {
  if [[ -n "$OSD_DEVICE" ]]; then
    if [[ -b "$OSD_DEVICE" ]]; then
      if [ -n "$BLUESTORE_BLOCK_SIZE" ]; then
        size=$(parse_size "$BLUESTORE_BLOCK_SIZE")
        if ! grep -qE "bluestore_block_size = $size" /etc/ceph/"${CLUSTER}".conf; then
          echo "bluestore_block_size = $size" >> /etc/ceph/"${CLUSTER}".conf
        fi
      fi
    else
      log "Invalid $OSD_DEVICE, only block device is supported"
      exit 1
    fi
  fi

  if [ ! -e "$OSD_PATH"/keyring ]; then
    if ! grep -qE "osd data = $OSD_PATH" /etc/ceph/"${CLUSTER}".conf; then
      echo "osd data = $OSD_PATH" >> /etc/ceph/"${CLUSTER}".conf
      echo "osd objectstore = bluestore" >> /etc/ceph/"${CLUSTER}".conf
    fi

    # bootstrap OSD
    mkdir -p "$OSD_PATH"
    chown --verbose -R ceph. "$OSD_PATH"

    # if $OSD_DEVICE exists we deploy with ceph-volume
    if [[ -n "$OSD_DEVICE" ]]; then
      ceph-volume lvm prepare --data "$OSD_DEVICE"
    else
      # we go for a 'manual' bootstrap
      ceph "${CLI_OPTS[@]}" auth get-or-create osd."$OSD_ID" mon 'allow profile osd' osd 'allow *' mgr 'allow profile osd' -o "$OSD_PATH"/keyring
      ceph-osd --conf /etc/ceph/"${CLUSTER}".conf --osd-data "$OSD_PATH" --mkfs -i "$OSD_ID"
    fi
  fi

  # activate OSD
  if [[ -n "$OSD_DEVICE" ]]; then
    OSD_FSID="$(ceph-volume lvm list --format json | python -c "import sys, json; print(json.load(sys.stdin)[\"$OSD_ID\"][0][\"tags\"][\"ceph.osd_fsid\"])")"
    ceph-volume lvm activate --no-systemd --bluestore "${OSD_ID}" "${OSD_FSID}"
  fi

When OSD_DEVICE exists then run ceph-volume.

memory tuning support

Implement this ceph/ceph-container#1283.

Use beast backend for rgw

Change https://github.com/ceph/cn-core/blob/master/cmd/init_mon.go#L72 to "beast".

Ability to connect multiple nanos

The idea is to be able to expend from one to many. Meaning that under the hood we have to configure the rgw replication. As a result you will have multiple gateways running configured to replicate in an active-active fashion.

cn-core fails to bootstrap in 512MB RAM environment

Upon bootstrapping ceph via cn-core in a containerized environment with 512MB RAM, cn-core throws an error stating 'init: run me with at least 512mb of ram' and fails.

Shrink ceph.conf as much as possible

The goal is to remove the ceph.conf.
Instead, pass all the required flag in the CLI.

Daemon startup wrong args

ceph        40  0.5  0.1 571984 125820 ?       Ssl  13:52   1:06 ceph-mon --setuser ceph --setgroup ceph -i ceph-nano-coree-faa32aebf00b --mon-data /var/lib/ceph/mon/ceph-ceph-nano-coree-faa32aebf00b --public-addr 127.0.0.1:3300
ceph       144  0.7  0.2 753936 149424 ?       Ssl  13:52   1:23 ceph-mgr --setuser ceph --setgroup -i ceph-nano-coree-faa32aebf00b
ceph       349  0.3  0.1 850272 69172 ?        Ssl  13:52   0:38 ceph-osd --setuser ceph --setgroup -i 0

mgr and osd are incorrect as they don't have the --setgroup flag set.
Also the osd mkfs does not have these flags.

Add more rgw hostnames

See: https://gist.github.com/robbat2/ec0a66eed28e5f0e1ef7018e9c77910c#rgw-zonegroup-configuration

We need to add $(hostname) and localhost to that list.

This means removing the rgw_dns_name entry from the ceph.conf I believe.