Multiple HAProxy Instances Started on Deployment about bamboo

[j1n6]

If it works as expected, Bamboo should handle those heavy repetitive events firing by Marathon.

Bamboo checks HAProxy configuration changes and only executes reload when necessary. In logs, you will see something like changes detected or no change.

Marathon triggers change event event when task state has no change. This is a known bug, as for as I know , they might plan a release at 0.7.1

On 26 Sep 2014, at 03:39, Dan Everton [email protected] wrote:

Using Mesos 0.20.1 and Marathon 0.7.0 I commonly see multiple HAProxy instances after a deployment event. It looks like Marathon doesn't batch events so Bamboo ends up restarting HAProxy multiple times in rapid succession. This leads to HAProxy not being able properly reload and thus multiple instances.

—
Reply to this email directly or view it on GitHub.

[j1n6]

Are you on bamboo version 0.2.2?

[deverton]

Bamboo is logging that a change has occurred. Those changes just happen to be milliseconds apart. I don't have the logs right now but I'll add them when I can. I'm using 0.22 of Bamboo.

[deverton]

Sorry, some more details. This seems to be happening because I'm experimenting with scaling an application group up and down a lot so there's quite a few events happening.

[j1n6]

I understand the issue now. We need to add reload delay or debounce on heavy reload events when scaling up/down happening too frequently.

[sttts]

I think the correct behaviour here would be:

PIDS=$(cat /var/run/haproxy.pid)
haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS
while ps -p $PIDS; do
  sleep 1
done

Of course, properly implemented in the go code around the current exec.Command line.