gluster / glusterd2

[DEPRECATED] Glusterd2 is the distributed management framework to be used for GlusterFS.

GlusterD-2.0

GlusterD-2.0 (GD2) is a re-implementation of GlusterD. It attempts to have better consistency, scalability and performance when compared with the current GlusterD, while also becoming more modular and easing extensibility.

Documentation

• User documentation
• Developer documentation

Architecture and Design

Please refer to the wiki for more information.

Building

To build GD2, just run make. If you don't have the required tools installed, run scripts/install-reqs.sh.

Contributing

We use the Github pull-request model for accepting contributions. If you are not familiar with the pull request model please read "Using pull requests". For specific information on GlusterD-2.0, refer the Development Guide.

Copyright and License

Copyright (c) 2015 Red Hat, Inc. http://www.redhat.com

This program is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License, version 3 or any later version (LGPLv3 or later), or the GNU General Public License, version 2 (GPLv2), in all cases as published by the Free Software Foundation.

glusterd2's Issues

Update contribution guidelines

As we've come back to Github pull-requests for contribution (from gerrithub), the documentation needs to be updated.

Add rest endpoints for peer commands

This also requires description of the ReST api for peer commands

Start etcd daemon upon starting of glusterd

Currently upon starting of glusterd daemon, etcd is not starting. For etcd bootstrapping first requirement is to run etcd daemon on every node.

Fix volume delete workflow

deleting a volume throws an error saying volume not found even if the volume exists

Following heketi APIs

License?

I have a bug fix I was going to contribute but I didn't see a license within the project. Can you please add license information so I can be explicit about it when I submit the pull request?

etcd proxy (client) support

Given a add peer request with proxy = true etcd should act as a client on that node, default shoud be to add etcd node as server

peer status should display the state of the peer

Add connected flag in peer object to indicate the health of the given peer

Enable unit tests for volume package by mocking out few syscalls

local state directory default path should be changed

Review volgen implementation

The first volgen implementation is based on the current C implementation in GlusterFS-3. Though this works, it is not suitable for the modular structure we want for GlusterD-2.0.

Document firewall/iptables port configuration for glusterd2

Add design spec

We need to add one quickly.

Refactor JSON response

Wrapping up other items along with the actual data is not something what other APIs follow. We should send the response data only for positive response.

etcd bootstrapping in GlusterD 2.0

Currently Glusterd2 is not doing etcd bootstrapping automatically. It should do member add/delete and bootstrap cluster accordingly upon execution of peer probe/peer detach command.

Volume name should be parameter instead of request body

In Create Volume,
name is sent in request body. For uniformity with Stop/Delete calls we can have volume name as parameter.

Create POST /volumes/{name}
Delete DELETE /volumes/{name}
Stop POST /volumes/{name}/stop
Start POST /volumes/{name}/start

Add unit tests for volume package

Consider having etcd cluster not managed and controlled by glusterd

Currently, the life-cycle of etcd idaemons are strictly coupled and controlled by glusterd2 process which makes the entire setup very fragile IMHO. Is it a good idea for glusterd2 to just use existing etcd cluster which is externally managed ?

Refactor volume commands

Currently individual volume commands are written in separate packages which is not ideal. Follow peer commands code to refactor.

Response handling in volgen failures

Current form of volgen code just panics if something goes wrong. We would need to send proper JSON responses in the case of failures.

Could not able to create volume if brick is having in hierarchical directory stracture

Currently volume creation fails if while volume creation brick are falling in hierarchical directory and that hierarchical directory does not exist.

Remove heketi dependency

heketi is pulled in as a dependency for tests, just to make use of an Assert function.

Fix test in TestIsETCDStartNeeded

Document the need for using clock synchronisation service such as NTP

When new peers are added to cluster, one should make sure that the time difference in time between peers in cluster is less than one second. If not, you'd see console/logs flooded with following warning

2016-07-14 14:23:36.414119 W | rafthttp: the clock difference against peer 1097c478b6d4774f is too high [29.889521687s > 1s]

Decide on store and remove libkv

Though libkv is helpful, it brings in a lot of dependencies which are not actually useful. Ref 1

It'll also be helpful if we decide on a particular store as we can make better use of its complete API.

Use etcd client APIs instead of libkv

Use uniform generate uuid function

volume create to work on a single node cluster

ReST API document

Need to create a document describing the ReST API for GlusterD-2.0

remove node from etcd cluster

As part of peer detach flow the same peer should be removed from etcd cluster.

Add etcd logging

etcd errors should be captured in some log file as otherwise it'd be difficult to understand the failures from etcd cluster.

Daemonize etcd

etcd instance can't be brought down when glusterd goes down since the other consumers like NSR would still be able to access it. This task is to have etcd daemonized such that on glusterd shutdown etcd instance doesn't go down.

Include Brick Online status in GET /volumes/{name}

Please include Brick Online status also in GET /volumes/{name}

Example,

"bricks": [
    {
        "id":"e5c26603-82bf-11e5-9d59-3c970e9eb10d",
        "hostname":"x.x.x.x"
        "path":  "/gluster/brick1",
        "online": true,
    },
    {
        "id":"06c47cc3-82c0-11e5-9d59-3c970e9eb10d",
        "hostname":"y.y.y.y"
        "path":  "/gluster/brick2",
        "online": false
    }
]

glusterd2 conf files are looked for multiple times during startup

Here's an excerpt from strace:

strace -f -o trace.txt -e 'trace=!mmap,rt_sigaction,futex,rt_sigprocmask,brk' glusterd2

1591  stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
1591  stat("/home/ppai", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
1591  stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
1591  stat("/home/ppai", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
1591  stat("/etc/glusterd/glusterd.json", 0xc820105078) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.toml", 0xc820105148) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.yaml", 0xc820105218) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.yml", 0xc8201052e8) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.properties", 0xc8201053b8) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.props", 0xc820105488) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.prop", 0xc820105558) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.hcl", 0xc820105628) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.json", 0xc8201056f8) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.toml", 0xc8201057c8) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.yaml", 0xc820105898) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.yml", 0xc820105968) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.properties", 0xc820105a38) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.props", 0xc820105b08) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.prop", 0xc820105bd8) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.hcl", 0xc820105ca8) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.json", 0xc820105d78) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.toml", 0xc820105e48) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.yaml", 0xc820105f18) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.yml", 0xc820128038) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.properties", 0xc820128108) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.props", 0xc8201281d8) = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.prop",  <unfinished ...>
1589  <... select resumed> )            = 0 (Timeout)
1589  select(0, NULL, NULL, NULL, {0, 20} <unfinished ...>
1591  <... stat resumed> 0xc8201282a8)  = -1 ENOENT (No such file or directory)
1591  stat("/etc/glusterd/glusterd.hcl", 0xc820128378) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.json", 0xc820128448) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.toml", 0xc820128518) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.yaml", 0xc8201285e8) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.yml", 0xc8201286b8) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.properties", 0xc820128788) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.props", 0xc820128858) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.prop", 0xc820128928) = -1 ENOENT (No such file or directory)
1591  stat("/home/ppai/glusterd.hcl", 0xc8201289f8) = -1 ENOENT (No such file or directory)

Validation, Commit & Rollback phases in volume create command life cycle

Fix transfer issues

I accidently transferred the repo to the Gluster organization. This has broken several things. All of which need to be fixed.

• Fix import paths for packages kshlm/glusterd2 -> gluster/glusterd2
• Fix wercker
  • Update readme
• Fix waffle
  • Create new board gluster/glusterd2
  • Update readme

Adding current node as a peer fails with etcd crash

If a node tries to add itself as a peer, it fails with error=client: etcd cluster is unavailable or misconfigured member=gd2-2 and etcd becomes defunct.

Fix regressions in volume create

volume create fails with no such file or directory error
volume info fails with 404 page not found

Add API Versioning

Cross GlusterD interaction using pbrpc

Multi node transaction framework support

Generic JSON response for all basic commands

Add document on how to add rest API end points

Inbuilt GlusterFlow v2 exporter

Do you reckon it would be much extra work to add a (hopefully simple) exporter for GlusterFlow v2? (to be built using InfluxDB, which is also go 😉)

All it would need to do is send the details of each client request to an InfluxDB database. (from each of the GlusterD daemons) Without knowing better, that doesn't sound like a huge amount of code. 😀

The benefit of that (especially for development) is being able to diagnose/understand (and maybe replay?) exactly what operations occurred from each member of the cluster, in which order. With accurate timestamps.

Add makefile

Network interface used by etcd should be configurable

I'm using a two node setup. Each node has 3 interfaces - one NAT, one host-only n/w and one loopback.

enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.2.15  netmask 255.255.255.0  broadcast 10.0.2.255
        inet6 fe80::a287:1bbe:d8fb:2283  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:4f:5d:32  txqueuelen 1000  (Ethernet)
        RX packets 7879  bytes 10958031 (10.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 971  bytes 70333 (68.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp0s8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.56.25  netmask 255.255.255.0  broadcast 192.168.56.255
        inet6 fe80::8599:e3a4:3bce:277b  prefixlen 64  scopeid 0x20<link>
        ether 08:00:27:6a:53:bd  txqueuelen 1000  (Ethernet)
        RX packets 24928  bytes 2030813 (1.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 23577  bytes 8516532 (8.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 79327  bytes 15173550 (14.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 79327  bytes 15173550 (14.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

etcd binary is spawned by passing IP address that etcd will use for communication. This IP passed is arrived at by GetLocalIP() method in utils/utils.go. In case of multiple network interfaces, this will return first IP it gets which is not loopback/localhost. This may not be desirable in many situations.

Get volume request should only display list of volume name and associated volume id of that volume

Description of problem:
Currently, when you make a volume Get request then it will display list of whole volume info (volinfo) object.

How reproducible:
Every time when you issue Get volume request.

Expected behaviour:
It should show list of volume name and associated volume id with it.

Initialize the pbrpc server/client/transport required for the transaction framework

Install etcd as a package requirement for GlusterD 2.0

Install etcd while executing install-reqs.sh (all necessary package for GlusterD 2.0).

Usage of force option in Create Volume

Single option to bypass multiple things is not good. For example, force option bypasses the following

1. Use the bricks used by other Volume
2. Allow creating in root dir

A user gets first error that bricks are used by other volume and uses force option to bypass that. User will not get any clue about second error even if its exists.

There should be separate flags for each bypass item also different option in CLI

For example,

"flags": {
    "reuse_bricks": true,
    "allow_root_dir": true
}

and in CLI

gluster volume create ... --allow-root-dir --reuse-bricks