costela / docker-volume-hetzner

Docker Volume Plugin for accessing Hetzner Cloud Volumes

Docker Volume Plugin for Hetzner Cloud

This plugin manages docker volumes using Hetzner Cloud's volumes.

This plugin is still in ALPHA; use at your own risk

Installation

To install the plugin, run the following command:

$ docker plugin install --alias hetzner ghcr.io/costela/docker-volume-hetzner:...-amd64

When using Docker Swarm, this should be done on all nodes in the cluster.

Important: the plugin expects the Docker node's hostname to match with the name of the server created on Hetzner Cloud. This should usually be the case, unless explicitly changed.

Plugin privileges

During installation, you will be prompted to accept the plugins's privilege requirements. The following are required:

• network: used for communicating with the Hetzner Cloud API
• mount[/dev/]: needed for accessing the Hetzner Cloud Volumes (made available to the host as a SCSI device)
• allow-all-devices: actually enable access to the volume devices mentioned above (since the devices cannot be known a priori)
• capabilities[CAP_SYS_ADMIN,CAP_CHOWN]: needed for running mount and chown

Usage

First, create an API key from the Hetzner Cloud console and save it temporarily.

Install the plugin as described above. Then, set the API key in the plugin options, where <apikey> is the key you just created:

$ docker plugin disable hetzner
$ docker plugin set hetzner apikey=<apikey>
$ docker plugin enable hetzner

Again, when using Docker Swarm, this should be done on all nodes in the cluster.

The plugin is then ready to be used, e.g. in a docker-compose file, by setting the driver option on the docker volume definition (assuming the alias hetzner passed during installation above).

For example, when using the following docker-compose volume definition in a project called foo:

volumes:
  somevolume:
    driver: hetzner

This will initialize a Hetzner volume named docker-foo_somevolume (see the prefix configuration below).

If the volume docker-foo_somevolume does not exist in the Hetzner Cloud project, the plugin will do the following:

1. Create the Hetzner Cloud (HC) volume
2. Attach the created HC volume to the node requesting the creation (when using docker swarm, this will be the manager node being used)
3. Format the HC volume (using fstype option; see below)
4. chown the volume to the appropriate uid/gid if specified.

The plugin will then mount the volume on the node running its parent service, if any.

Configuration

The following options can be passed to the plugin via docker plugin set (all names case-sensitive):

• apikey (required): authentication token to use when accessing the Hetzner Cloud API
• size (optional): size of the volume in GB (default: 10)
• fstype (optional): filesystem type to be created on new volumes. Currently supported values are ext{2,3,4} and xfs (default: ext4)
• prefix (optional): prefix to use when naming created volumes; the final name on the HC side will be of the form prefix-name, where name is the volume name assigned by docker (default: docker)
• loglevel (optional): the amount of information that will be output by the plugin. Accepts any value supported by logrus (i.e.: fatal, error, warn, info and debug; default: warn)
• use_protection (optional): whether to enable/disable deletion protection on creation/deletion. Disable this if you want to manage deletion protection yourself. (default: true)
• uid (optional): which user id to use by default as owners for the filesystem of newly created volumes
• gid (optional): which group id to use by default as owners for the filesystem of newly created volumes

Additionally, size, fstype, uid and gid can also be passed as options to the driver via driver_opts:

volumes:
  somevolume:
    driver: hetzner
    driver_opts:
      size: '42'
      fstype: xfs
      uid: '999'
      gid: '999'

⚠️ Passing any option besides size, fstype, uid and gid to the volume definition will have no effect beyond a warning in the logs. Use docker plugin set instead.

Limitations

• Concurrent use: Hetzner Cloud volumes currently cannot be attached to multiple nodes, so the same limitation applies to the docker volumes using them. This also precludes concurrent use by multiple containers on the same node, since there is currently no way to enforce docker swarm services to be managed together (cf. kubernetes pods).
• Single location: since volumes are currently bound to the location they were created in, this plugin will not be able to reattach a volume if you have a swarm cluster across locations and its service migrates over the location boundary.
• Volume resizing: docker has no support for updating volume definitions. After a volume is created, its size option is currently ignored. This may be worked around in a future release.
• Docker partitions: when used in a docker swarm setup, there is a chance a network hiccup between docker nodes might be seen as a node down, in which case the scheduler will start the container on a different node and will "steal" its volume while in use, potentially causing data loss.

docker-volume-hetzner's Issues

Use with existing volume?

Thanks for this plugin, I think it could come in very useful! 🎉

I have an existing volume, let's call it volume-nbg1-1-test, with the ID 1234567.

How would I go about using this volume and attaching it to a Docker container?

I tried a simple docker-compose.yml with:

version: "3.5"

volumes:
    volume-nbg1-1-test:
        driver: hetzner

But it returns:

Creating volume "docker-volume-test_volume-nbg1-1-test" with hetzner driver

So it's adding some prefix, and I assume it's not looking for existing volumes?

arm64 version

As Hetzner has arm64 instances, it would be great if we could add arm64 to CSI here.

more info here hetznercloud/csi-driver#428 (Hetzner CSI driver in beta)

Add a reference to podlike?

Something I stumbled across that might be of interest to some people using this plugin:

https://github.com/rycus86

This simulates the concept of Pods in Docker Swarm. While maybe not something to use for everything, this would help for the case where some docker containers really want to share a volume to talk to each other.

Thanks for creating this

Hey,
I know a GitHub issue is maybe not the correct place. But I wanted to say thank you for creating this awesome plugin. It works really great.

Best wishes,
Timo

"unable to authenticate (unauthorized)" error

Trying to set up a new volume:

➜ cat docker-compose.yml 
version: "3.5"

volumes:
    test:
        driver: hetzner
        driver_opts:
            apikey: xxxx
            fstype: xfs
            size: 10 # in GB, default: 10
            loglevel: debug

When running docker-compose up, I get:

Creating volume "docker-volume-test_test" with hetzner driver
ERROR: create docker-volume-test_test: VolumeDriver.Create: could not get cloud server 'test': unable to authenticate (unauthorized)

Note that the token is correct; I could successfully activate the context in hcloud and run:

➜ ~/go/bin/hcloud server describe test
ID:     xxxxxxxx
Name:       test
Status:     running
Created:    Wed Dec 11 09:00:43 CET 2019 (5 hours ago)
Server Type:    cx41 (ID: 7)
  ID:       7
  Name:     cx41
  Description:  CX41
  Cores:    4
  Memory:   16 GB
  Disk:     160 GB
  Storage Type: local
Public Net:
  IPv4:
    IP:     xx.xx.xx.xx
    Blocked:    no
    DNS:    static.xx.xx.xx.xx.clients.your-server.de
  IPv6:
    IP:     xx.xx.xx.xx
    Blocked:    no
  Floating IPs:
    No Floating IPs
Private Net:
  - ID:         17489
    Name:       internal-network
    IP:         10.0.0.4
    MAC Address:    86:00:00:39:19:92
    Alias IPs:      -
Volumes:
  No Volumes
Image:
  ID:       168855
  Type:     system
  Status:   available
  Name:     ubuntu-18.04
  Description:  Ubuntu 18.04
  Image size:   -
  Disk size:    5 GB
  Created:  Wed May  2 13:02:30 CEST 2018 (2 years ago)
  OS flavor:    ubuntu
  OS version:   18.04
  Rapid deploy: yes
Datacenter:
  ID:       2
  Name:     nbg1-dc3
  Description:  Nuremberg 1 DC 3
  Location:
    Name:       nbg1
    Description:    Nuremberg DC Park 1
    Country:        DE
    City:       Nuremberg
    Latitude:       49.452102
    Longitude:      11.076665
Traffic:
  Outgoing: 7.4 MB
  Ingoing:  2.4 GB
  Included: 22 TB
Backup Window:  Backups disabled
Rescue System:  disabled
ISO:
  No ISO attached
Protection:
  Delete:   no
  Rebuild:  no
Labels:
  No labels

Any idea what's wrong? When I run hostname, I also get test.

size option is different for some volumes

Hi,

thanks for this driver it makes things a lot easier.

It was working for a while very well but since some days this errors shows up for some of my volumes, not all:

Configuration for volume XXXX specifies "size" driver_opt 100, but a volume with the same name uses a different "size" driver_opt (None). If you wish to use the new configuration, please remove the existing volume "XXXX"

It seems like docker looses the size of the volume:

docker volume inspect XXXXX
[
{
"CreatedAt": "2021-01-20T06:52:53Z",
"Driver": "hetzner:latest",
"Labels": null,
"Mountpoint": "/mnt/649e5e2a28e1ad8fe3edee280141f3480f98cbd36a3e2d117bdec2ed5be00f06",
"Name": "XXXX",
"Options": null,
"Scope": "global",
"Status": {
"mounted": true
}
}
]

How can i fix this? Does somebody have similar problems?

Thanks,

Jannis

Improvement: Add Backoff for Rate Limits

Currently it can happen that for whatever reason the requests to the Hetzner Cloud API are going a bit crazy, causing 429 errors.

Currently, once a rate limit is reached, we keep hammering the rate API and we never get out of the rate limit issue.

Feature idea: softdelete volumes

Something to consider as a safety measure: Instead of deleting the Volumes on Hetzner we could add a Mode to the plugin where instead of deleting Volumes we simply hide them either via some Label "deleted" or by renaming Volumes.

Leaving this here as something to think about. Unsure whether this is actually needed, but worth a thought.

"VolumeDriver.Mount: could not mount

Hello! I try to connect the volume to the working node of the swarm. Gives such an error. Help me to understand.

Shutdown Rejected less than a second ago "VolumeDriver.Mount: could not mount '/dev/disk/by-id/scsi-0HC_Volume_3651104' as any of [ext4 xfs ext3 ext2]"

redis-data:
    driver: hetzner
    driver_opts:
      size: '10'

Add support for automatic resizing if volume size was expanded directly on Hetzner

A stopgap solution until a proper workaround has been found for expanding the volumes could be to add an additional step to mounting where we could check the size of the volume when we created it vs what it is right now.

Then, if it differs, we could run resize2fs before mounting it to the actual container.

What do you think?

Can't create or mount volumes

While trying to test this, i cannot create or mount volumes.

$ docker volume create -d hetzner test
Error response from daemon: create test: VolumeDriver.Create: EOF

The plugin seems to be configures correctly, because the ls subcommand lists already existing hetzner volumes:

$ docker volume ls
DRIVER              VOLUME NAME
hetzner:latest      postgres_test

The log:

Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="2019/08/14 09:22:45 Entering go-plugins-helpers capabilitiesPath" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="2019/08/14 09:22:45 Entering go-plugins-helpers getPath" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="2019/08/14 09:22:45 Entering go-plugins-helpers createPath" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="2019/08/14 09:22:45 http: panic serving @: runtime error: invalid memory address or nil pointer dereference" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="goroutine 44 [running]:" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="net/http.(*conn).serve.func1(0xc00025abe0)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/net/http/server.go:1746 +0xd0" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="panic(0x6b5440, 0x9396f0)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/runtime/panic.go:513 +0x1b9" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="main.(*hetznerDriver).Create(0xc00000c040, 0xc0003026c0, 0xc000368400, 0x68dd00)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/plugin/driver.go:57 +0x1c5" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="github.com/docker/go-plugins-helpers/volume.(*Handler).initMux.func1(0x760cc0, 0xc0003909a0, 0xc000368400)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/go/pkg/mod/github.com/docker/[email protected]/volume/api.go:139 +0xe9" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="net/http.HandlerFunc.ServeHTTP(0xc00000ec60, 0x760cc0, 0xc0003909a0, 0xc000368400)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/net/http/server.go:1964 +0x44" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="net/http.(*ServeMux).ServeHTTP(0xc00004ede0, 0x760cc0, 0xc0003909a0, 0xc000368400)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/net/http/server.go:2361 +0x127" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="net/http.serverHandler.ServeHTTP(0xc000050f70, 0x760cc0, 0xc0003909a0, 0xc000368400)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/net/http/server.go:2741 +0xab" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="net/http.(*conn).serve(0xc00025abe0, 0x760e80, 0xc00003e700)" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/net/http/server.go:1847 +0x646" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="created by net/http.(*Server).Serve" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45+02:00" level=error msg="\t/usr/local/go/src/net/http/server.go:2851 +0x2f5" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:45 dockerhost dockerd[7173]: time="2019-08-14T11:22:45.884762170+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f/hetzner.sock/VolumeDriver.Create: Post http://%2Frun%2Fdocker%2Fplugins%2F8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f%2Fhetzner.sock/VolumeDriver.Create: EOF, retrying in 1s"
Aug 14 11:22:46 dockerhost dockerd[7173]: time="2019-08-14T11:22:46+02:00" level=error msg="2019/08/14 09:22:46 Entering go-plugins-helpers createPath" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:46 dockerhost dockerd[7173]: time="2019-08-14T11:22:46+02:00" level=error msg="2019/08/14 09:22:46 Entering go-plugins-helpers capabilitiesPath" plugin=8065d7c1773187ab2e9c1dcee0e2e717c0966cd7c795695c6716250cc2a27f3f
Aug 14 11:22:46 dockerhost dockerd[7173]: time="2019-08-14T11:22:46.887388478+02:00" level=error msg="Handler for POST /v1.40/volumes/create returned error: create testvolume: VolumeDriver.Create: EOF\n"

This is on ubuntu 18.04:

$ docker --version
Docker version 19.03.1, build 74b1e89

Add plumbing for Volume Labels

We are building infrastructure for multitenancy around Docker Swarm using labels on objects - see https://github.com/neuroforgede/swarmgate

I would love for this volume plugin to set and get labels from the Hetzner API. This way I can properly use our multi-tenancy concept with the driver.

What do you think @costela ? Should be possible as far as I can tell.

https://github.com/moby/moby/blob/93fffa299c2cb23055bb54c380a02d53c9a7d525/daemon/cluster/executor/container/container.go#L444

https://github.com/moby/moby/blob/93fffa299c2cb23055bb54c380a02d53c9a7d525/volume/drivers/proxy.go#L26

Unable to connect to plugin

Hi,

thx for the plugin!
I'm trying to use it with docker swarm, but can't get it to work.
I set it up like described in the readme.
This is my exemplatory docker-compose file for the stack:

version: '3.7'
services:
  mysql:
    image: mysql:5.7
    volumes:
      - mysql-data:/var/lib/mysql/data

volumes:
  mysql-data:
    driver: hetzner

when i deploy my stack using

docker stack deploy -c mysql.yml mysql

the docker logs show:

Unable to connect to plugin: /run/docker/plugins/bf53cfb6af1533728ae02cf7768a851edcf83d725a445852381a35e3dd4de7d4/hetzner.sock/VolumeDriver.Create: Post http://%2Frun%2Fdocker%2Fplugins%2Fbf53cfb6af1533728ae02cf7768a851edcf83d725a445852381a35e3dd4de7d4%2Fhetzner.sock/VolumeDriver.Create: EOF, retrying in 1s

and the container doesn't start. Due to the error stating VolumeDriver.Create i tried to create the volume myself and use it, but then the logs show:

Dec 21 10:10:49 116 dockerd[642]: time="2019-12-21T10:10:49.836744136+01:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/bf53cfb6af1533728ae02cf7768a851edcf83d725a445852381a35e3dd4de7d4/hetzner.sock/VolumeDriver.Mount: Post http://%2Frun%2Fdocker%2Fplugins%2Fbf53cfb6af1533728ae02cf7768a851edcf83d725a445852381a35e3dd4de7d4%2Fhetzner.sock/VolumeDriver.Mount: EOF, retrying in 1s"

My docker version is 19.03.5

Thx a lot.

madurity stage

Hi

first of all, good jobs, this projects fits perfectly with me needed, i see that have more than 3 years with many commits, but in the readme we can found that is in alpha stage. Do you thing that it isn't ready for production yet?

My idea is use it for store the data of a mariadb deployed in a swarm cluster in hetzner vps and achieve application level high aviability

thanks.

Error during docker-compose start: VolumeDriver.Mount: could not attach volume

Hi there,

I wanted to use your plugin, but when I run a docker-compose setup, I get the following error:

ERROR: for web Cannot start service web: error while mounting volume '/var/lib/docker/plugins/******/propagated-mount/******': VolumeDriver.Mount: could not attach volume 'docker-nextcloud-dockerized_nextcloud' to 'cloud': cannot perform operation because server is locked (locked)

I think this happens because I have two services defined that access the same volume, though one mounts it read-only.

The set-up is the exact nextcloud docker-compose setup: https://github.com/nextcloud/docker/blob/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/fpm/docker-compose.yml

Could you support me in getting this to work?

Use with replicas

Is it safe to use this with a Swarm / Compose service that has multiple replicas? Expected behaviour would be to create a volume named for the replica number of the container.

change permissions of newly created volumes to uid:gid

Hey there,

I was wondering if it was possible in concept to automatically chown a newly created volumes to a specific user. While generally being awesome, we have to change containers to run as root sometimes just to automatically be able to create folder structures.

If you think it is generally possible, I'd be happy to send a PR your way :)