drone / autoscaler Goto Github PK

needed to tag a quick release for a bug fix, without delay. the release notes are in progress (we merged a ton of pull requests) and are coming soon. thank you for your patience!

It looks like AWS spot instance is supported (https://github.com/search?q=repo%3Adrone%2Fautoscaler%20spot&type=code) but not GCP yet? Thanks!

This is a huge cost saving for us since spot instance is ~5x cheaper

Since today, the autoscaler is unable to provision new instances on hetzner cloud. The drone cli tells me the ubuntu 16.04 image could not be found:

$ drone server info $(drone server ls)
Name: agent-wawR6snj
Address: 
Region:  
Size:    
State:   error
Error:   image 'ubuntu-16.04' not found (invalid_input)

My guess is hetzner deprecated the ubuntu 16.04 image.

I was able to fix this by setting the DRONE_HETZNERCLOUD_IMAGE environment variable to ubuntu-20.04 - I think the default should be set to 20.04. I could send a PR if you'd accept one.

Documentation says that DRONE_HETZNERCLOUD_SSHKEY must be an integer. How does such integer represent an SSH key?
I already added the public key to Hetzner, but how do I provide the private key to the autoscaler and how does an SSH key relate to a simple integer?

It seems like the capacity number for each server is always the number of cores and not configurable?

Would it make sense to have an option similar to DRONE_RUNNER_CAPACITY for autoscaler?

Once in a while the created instance on Hetzner does not get destroyed. This results in a big bill from Hetzner because the server is still running.

Can it be that the autoscaler forgets that some servers exist when the autoscaler gets restarted, or the git server gets restarted?

This is our config:

drone-autoscaler:
    image: drone/autoscaler:1.8.2
    restart: unless-stopped
    volumes:
      - drone_autoscaler_data:/data
    environment:
      - DRONE_POOL_MIN=0
      - DRONE_POOL_MAX=4
      - DRONE_POOL_MIN_AGE=1h
      - DRONE_CAPACITY_BUFFER=0
      - DRONE_AGENT_CONCURRENCY=5
      - DRONE_SERVER_PROTO=https
      - DRONE_SERVER_HOST=REDACTED
      - DRONE_SERVER_TOKEN=${DRONE_SERVER_TOKEN}
      - DRONE_AGENT_TOKEN=${DRONE_RPC_SECRET}
      - DRONE_HETZNERCLOUD_DATACENTER=${DRONE_HETZNERCLOUD_DATACENTER}
      - DRONE_HETZNERCLOUD_IMAGE=ubuntu-20.04
      - DRONE_HETZNERCLOUD_TYPE=cx51
      - DRONE_HETZNERCLOUD_SSHKEY=${DRONE_HETZNERCLOUD_SSHKEY}
      - DRONE_HETZNERCLOUD_TOKEN=${DRONE_HETZNERCLOUD_TOKEN}
      - DRONE_INTERVAL=10s
      - DRONE_LOGS_DEBUG=false

Running an autoscaler to test arm64 builds.
AWS EC2 m6g.large
Sporadically seeing failed builds on newly launched instances:

clone: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

latest: Pulling from drone/git
Digest: sha256:091ecd02ee4ac5154fd76133c5055b2345a61cbc17182b00612df1fa7eef1510
Status: Downloaded newer image for drone/git:latest

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

However, inot consistently, maybe 10% of the time. Most of the time, no errors.

What comes to mind is to put a "sleep 1" or "systemctl restart docker" someplace. A very small change.

It looks like that is already present "[ systemctl, restart, docker ]"

Now trying a modification in drivers/internal/userdata/userdata.go . Change

  - path: /etc/docker/daemon.json
    content: |
      {
        "hosts": [ "0.0.0.0:2376", "unix:///var/run/docker.sock" ],
        "tls": true,
        "tlsverify": true,
        "tlscacert": "/etc/docker/ca.pem",
        "tlscert": "/etc/docker/server-cert.pem",
        "tlskey": "/etc/docker/server-key.pem"
      }

to

  - path: /etc/docker/daemon.json
    content: |
      {
        "hosts": [ "0.0.0.0:2376", "unix:///var/run/docker.sock" ],
        "containerd": "/run/containerd/containerd.sock",
        "tls": true,
        "tlsverify": true,
        "tlscacert": "/etc/docker/ca.pem",
        "tlscert": "/etc/docker/server-cert.pem",
        "tlskey": "/etc/docker/server-key.pem"
      }

Adding "containerd": "/run/containerd/containerd.sock" , based on this article https://stackoverflow.com/questions/68823645/what-is-purpose-of-the-switch-containerd-in-command-dockerd

I'll report back, if it eventually improves the condition.

I would suggest to update the default for the used Watchtower image as DockerHub will switch the payment policy and I won't way the huge amount which will lead to the removal of the Webhippie organization on DockerHub. If you want to continue using this Watchtower image you should switch to ghcr.io/dockhippie/watchtower or quay.io/webhippie/watchtower.

Spent the last few hours grappling with my agent docker containers deployed via autoscaler not having port 3000 exposed. As a result, drone could push jobs to the agent. I dove the source code and found the ENV var mentioned in the subject. It would have saved a lot of time if this was documented at autoscale.drone.io. Not sure how to go about contributing, figured it was worth a mention.

Furthermore, why does this var not have a default of 3000:3000 in the code? is this by design? or an overlooked issue? figure I may be doing something wrong and would like confirmation before opening a PR to make that change.