Backed does not come back from dead status about pound

[graygnuorg]

I indeed have Control configured but never used poundctl. If you agree, you can close this issue as "not planned" while I'll be monitoring the problem and eventually get back to you.

Ok. If it occurs again, please be sure to include the output of "poundctl list" and a couple of lines from the log file, that follow the last connect_nb message (only warnings and errors are of interest, normal "access-style" logging is not). Regards, Sergey

[graygnuorg]

In the lack of further info, I'm closing that as completed.

[graygnuorg]

Is there anything we can check?

Yes, look for warning messages in pound log that begin with "connect_nb:". These should appear each 30 seconds (the default alive interval). Do you see any? Regards, Sergey

[abaldoni]

For that host, I can see them only twice, at 30 seconds from each other, but after those there are no more of them.

Regards,
Alessandro

[graygnuorg]

abaldoni ***@***.***> ha escrit:

For that host, I can see them only twice, at 30 seconds from each other, but after those there are no more of them.

What messages do you see?

[abaldoni]

Hello Sergey,
Sep 3 07:07:42 pound-101 pound[969]: (7fe30fca7640) connect_nb: error after getsockopt: Connection refused
Sep 3 07:08:12 pound-101 pound[969]: (7fe312ce5640) connect_nb: error after getsockopt: Connection refused

and after that timestamp no more.

[graygnuorg]

Thanks for additional info. Can you show me the "Service" section where this backend is defined? Is that the only backend that behaves this way?

[abaldoni]

Yes, it is (apparently) the only one.
We have other services that get restarted daily but the work fine.
The main difference is that this is a Tomcat on a Windows system and not on Linux.
The configuration is utterly simple:

        Service
                HeadRequire "Host: .*REDACTED*"
                URL ".*/XXX/YY/ZZZ.*"

                BackEnd
                        Address REDACTED
                        Port 8280
                End
        End

[graygnuorg]

The configuration is utterly simple:

Indeed. That's weird. Do you have "Control" configured? If so, have you examined the status of this backend using poundctl list?

[abaldoni]

I indeed have Control configured but never used poundctl.
If you agree, you can close this issue as "not planned" while I'll be monitoring the problem and eventually get back to you.

[nagyrobi]

I ran into the same issue recently. It was a linux backend. Had to restart pound for the fix.

[graygnuorg]

Thanks for the feedback. Can you give some more detail please: version of pound, number of backends in the service, anything else that you think might be of interest?

[nagyrobi]

In my case this is pound (pound) 4.6.90.
Have 9 backend services in total, each with one backend configured.
The affected part of the config:

ListenHTTP
	Service
		Host "my.domain.tld"
		Redirect 301 "https://my.domain.tld"
	End
End

Service
	Host "my.domain.tld"
	BackEnd
		Address 192.b.x.y
		Port    zzzzz
		WSTimeOut	60
	End
End

It happened that the host at 192.b.x.y (a qemu VM) was unproperly killed. After a reboot, pound didn't see it anymore.
Could be something tricky probably from the network side, as there was surely no proper announcement to the network that the host is going down. I only noticed after several days, that although the host is up and reacheable by is internal IP, it's not accessible through pound.

[graygnuorg]

Was there anything in the pound log? Things of interest are messages beginning with "connect_nb", or any error messages?

[graygnuorg]

After many attempts I had finally been able to reproduce this behavior. To make sure I'm on the right track, I need the following info:

1. Is pound running as daemon or foreground program (the Daemon configuration setting)?
2. The value of LogLevel setting
3. Command line options with which pound is started.

[nagyrobi]

1. No Daemon configuration specified in the config file (according to the doc, it runs as daemon as default)
2. LogLevel 0
3. no command line options, it's invoked by the rc.d script as /usr/local/sbin/pound

[graygnuorg]

Thanks. Then, I suppose, you should have seen several connect_nb: messages in your pound log, and then nothing up to the moment of pound restart. Is that so?

[nagyrobi]

Yep indeed, hundreds of them:

Time	Process	PID	Message
Sep 12 15:45:28	pound	9032	(800a79d00) connect_nb: error after getsockopt: Connection refused
Sep 12 15:45:13	pound	9032	(800a79d00) connect_nb: error after getsockopt: Connection refused
Sep 12 15:44:58	pound	9032	(800a79d00) connect_nb: error after getsockopt: Connection refused
Sep 12 15:44:43	pound	9032	(800a79d00) connect_nb: error after getsockopt: Connection refused
Sep 12 15:44:28	pound	9032	(800a79d00) connect_nb: error after getsockopt: Connection refused
...

[graygnuorg]

Ok. Last (hopefully) question: what was the time difference between last of them and the time when you restarted pound?

[abaldoni]

I have a (maybe) new insight: the backend that does not get resurrected is used multiple times in different listeners.
Today, I got the following status (grepped from poundctl):

    0. backend http 192.168.0.61:8280 5 alive active
    0. backend http 192.168.0.61:8280 5 alive active
    0. backend http 192.168.0.61:8280 5 dead active
    0. backend http 192.168.0.61:8280 5 alive active

As always I had to restart pound to get it 'alive' again. Using 'poundctl disable/enable' did not work.

[nagyrobi]

Ok. Last (hopefully) question: what was the time difference between last of them and the time when you restarted pound?

Didn't catch that info. Logs rotated since then...

But I guess it was continuous flood until restart.

[graygnuorg]

Hi Alessandro,

I have a (maybe) new insight: the backend that does not get resurrected is used multiple times in different listeners. Today, I got the following status (grepped from poundctl):

That's interesting, thanks for the info. As a matter of fact, for pound these are different backends, although having the same target address. Nevertheless, it may indeed be important. Were other backends (among these) operational while that one remained in the dead state? Pound marks the backend as dead only if there was an unsuccessful attempt to connect to it, so it is important to know if any requests were successfully served by these while the other one remained inactive. Another question: according to logs, what was the time difference between the moment you restarted pound and the most recent 'connect_nb:' message? Did pound serve any other requests in between? Regards, Sergey

[StomproLARL2023]

Hello, I think I'm also noticing this issue.

pound 4.9
Launched from systemd

One interesting thing is that the websocketd backend was redetected as alive once its server was brought back up, but the http and https services were not.

Set 3 backends for 3 services, https, http and websockets on port 7682.

Only had one live backend server but entries for the future ones were added. Only the active ones showed as alive. Brought up the new backend server and pound only detected that the websockd service was back up. http and https just showed "dead active".

Tried using poundctl to disable and enable them, but that didn't seem to help.

[abaldoni]

Yes, at least one of them responded to requests while the other was dead.
The web server hosting the backends is restarted every morning at about 7AM so the connect_nb message is always at that time, regardless of when I restart pound.

[graygnuorg]

Thanks for the feedback. Please try commit bfd8c1d.

[graygnuorg]

@abaldoni , @nagyrobi , Any news?

[nagyrobi]

Not yet, as I need to set up environment to compile under bsd.

[abaldoni]

I deployed the commit last saturday and, as of today, we experienced no problems at all. 👍