Logs about haproxy-consul-connect

[pierresouchay]

Logs can be sent directly to syslog. No, you don't need consul-template.

If you are interested, there is a presentation done a few months ago about features and architecture of project that has been put online recently: https://www.haproxy.com/user-spotlight-series/building-a-service-mesh-at-criteo-with-consul-and-haproxy/

[pvyaka01]

Yes i saw that project and thank you. I'm using Consul's dashboard and counting example - no docker or k8s. Here's what i noticed:

./haproxy-consul-connect -dataplane /opt/haproxy/bin/dataplaneapi -haproxy /opt/haproxy/bin/haproxy -sidecar-for counting-1 -stats-addr 0.0.0.0:1936 -stats-service-register
INFO[0000] consul: watching service counting-1
INFO[0000] consul: watching service counting-1-sidecar-proxy
INFO[0000] consul: CA certs changed, active root id: 98:c3:ff:ca:a7:1a:f2:a5:ed:0b:04:3a:80:2c:62:a8:ec:35:3c:64
INFO[0000] consul: CA certs ready
INFO[0000] consul: leaf cert for service counting changed, serial: 0e:16:b2, valid before: 2020-05-07 16:33:55 +0000 UTC, valid after: 2020-05-04 16:33:55 +0000 UTC
INFO[0000] consul: leaf cert for counting ready
INFO[0000] received consul config update
INFO[0000] handling new configuration
INFO[0000] haproxy: 126/020054 (17625) : New worker #1 (17630) forked
ERRO[0000] haproxy: time="2020-05-06T02:00:54Z" level=info msg="HAProxy Data Plane API v2.0.0 448d36e"
ERRO[0000] haproxy: time="2020-05-06T02:00:54Z" level=info msg="Build from: [email protected]:haproxytech/dataplaneapi.git"
ERRO[0000] haproxy: time="2020-05-06T02:00:54Z" level=info msg="Build date: 2020-04-27T15:28:42Z"
INFO[0000] Starting stats server at 0.0.0.0:1936
INFO[0000] state applied
WARN[0001] haproxy: 126/020055 (17625) : Reexecuting Master process
INFO[0001] haproxy: 126/020055 (17625) : New worker #1 (17639) forked
WARN[0001] haproxy: 126/020055 (17630) : Stopping frontend GLOBAL in 0 ms.
WARN[0001] haproxy: 126/020055 (17630) : Proxy GLOBAL stopped (FE: 1 conns, BE: 1 conns).
WARN[0002] haproxy: 126/020056 (17625) : Former worker #1 (17630) exited with code 0 (Exit)

and /var/log/messages do not show any http stats either:
May 6 02:02:54 consul: 2020-05-06T02:02:54.966Z [INFO] agent: Synced service: service=counting-1-connect-stats
May 6 02:03:54 consul[32095]: agent: Synced service: service=counting-1-connect-stats
May 6 02:03:54 consul: 2020-05-06T02:03:54.964Z [INFO] agent: Synced service: service=counting-1-connect-stats

I can hit the service and see prometheus style metrics with port :1936/metrics. No http access logs or stats page since in the generated tmp haproxy.conf stats are not enabled. I've an issue with intentions not being honored, will open a different issue about that.

HAProxy version: 2.0.9 (can't use latest due to other reasons)

And
./dataplaneapi --version
HAProxy Data Plane API v2.0.0 448d36e

Build from: [email protected]:haproxytech/dataplaneapi.git
Build date: 2020-04-27T15:28:42Z

Thanks for your help!

[pierresouchay]

In theory, logs from HAProxy are sent to the same log channel as logs from app and prefixed with haproxy. We can see some logs from HAProxy in the output you provided:

WARN[0001] haproxy: 126/020055 (17625) : Reexecuting Master process

You can tune the log level with the flag log-level (INFO) by default and all logs from HAProxy should be displayed (the binding with log level is here: https://github.com/haproxytech/haproxy-consul-connect/blob/master/haproxy/halog/log.go#L55 )

What are the logs you are expecting ?

[pvyaka01]

Sorry perhaps i'm not clear. I'm referring to httplog and tcplog directives. We rely on these quite heavily. We've more than 100 instances of haproxy across multiple environments.

When HAProxy is run as a Layer 7 proxy via mode http, you should add the option httplog directive. It ensures that HTTP requests and responses are analyzed in depth and that no RFC-compliant content will go uncaptured. This is the mode that really highlights the diagnostic value of HAProxy. The HTTP log format provides the same level of information as the TCP format, but with additional data specific to the HTTP protocol. If you were to re-create this format using log-format, it would look like this:

This is from haproxy docs.
https://www.haproxy.com/blog/introduction-to-haproxy-logging/

Also, stats page results in "404" http://:<stats_port>/stats but Prometheus metrics works on http://:<stats_port>/metrics

Thanks for the help!

[ShimmerGlass]

Log level trace will relay haproxy httplog to haproxy-consul-connect stderr. However please note this has a performance cost on high RPS workloads.

[pvyaka01]

Thanks will try that. HAProxy itself doesn't need tracing or even debug to generate httplog. I know this is different. Appreciate the help

[pvyaka01]

Command i used:
./haproxy-consul-connect -dataplane /opt/haproxy/bin/dataplaneapi -haproxy /opt/haproxy/bin/haproxy -sidecar-for counting-1 -stats-addr 0.0.0.0:1936 -stats-service-register -log-level trace

Log level trace does not work at all. Debug works but no http response/request logs like i mentioned before. When i use "-log-level trace" I see below messages:

DEBU[0015] sending dataplane req: POST /v2/services/haproxy/configuration/log_targets?parent_type=frontend&parent_name=front_downstream&transaction_id=16030995-6187-4b7d-be16-a16d4a987dde
ERRO[0015] error calling POST /v2/services/haproxy/configuration/log_targets?parent_type=frontend&parent_name=front_downstream&transaction_id=16030995-6187-4b7d-be16-a16d4a987dde: response was 422: "{"code":602,"message":"index in body is required"}"
DEBU[0016] sending dataplane req: GET /v2/services/haproxy/stats/native
DEBU[0017] sending dataplane req: GET /v2/services/haproxy/stats/native
DEBU[0018] sending dataplane req: GET /v2/services/haproxy/stats/native
INFO[0018] handling new configuration
DEBU[0018] applying new state: {Frontends:[{Frontend:{Clflog:false ClientTimeout:0x122de80 Clitcpka: Contstats: DefaultBackend:back_downstream Dontlognull: Forwardfor: HTTPUseHtx: HTTPConnectionMode: HTTPKeepAliveTimeout: HTTPRequestTimeout: Httplog:true LogFormat: LogFormatSd: LogSeparateErrors: LogTag: Maxconn: Mode:http Name:front_downstream Tcplog:false} Bind:{AcceptProxy:false Address:0.0.0.0 Alpn: Name:front_downstream_bind Port:0xc0005451a8 Process: Ssl:true SslCafile:/tmp/haproxy-connect-726206365/57f2efdcc7d285819fb28cab4b1cd6f09d816b0dd8b5a5f7e8a6d084145acb4d SslCertificate:/tmp/haproxy-connect-726206365/66d33951e328b31f04eabf9dfd06a9f811b87cf38f23b668e346fccaa29aa53b TCPUserTimeout: Transparent:false V4v6:false Verify:required} LogTarget:0xc000131500 Filter:}] Backends:[{Backend:{AdvCheck: Balance: CheckTimeout: ConnectTimeout:0x122de88 Cookie: DefaultServer: ExternalCheck: ExternalCheckCommand: ExternalCheckPath: Forwardfor: HTTPUseHtx: HTTPConnectionMode: HTTPKeepAliveTimeout: HTTPPretendKeepalive: HTTPRequestTimeout: Httpchk: LogTag: Mode:http Name:back_downstream QueueTimeout: Redispatch: Retries: ServerTimeout:0x122de90 StickTable:} LogTarget:0xc0001315e0 Servers:[{Address:127.0.0.1 Backup: Check: Cookie: Inter: Maintenance: Maxconn: Name:downstream_node OnError: OnMarkedDown: OnMarkedUp: Port:0xc0005451b8 Ssl: SslCafile: SslCertificate: TLSTickets: Verify: Weight:}]}]}
DEBU[0018] sending dataplane req: POST /v2/services/haproxy/transactions?version=1
DEBU[0018] sending dataplane req: POST /v2/services/haproxy/configuration/frontends?transaction_id=653419f0-0326-46fd-a7ce-cc528362b9bc
DEBU[0018] sending dataplane req: POST /v2/services/haproxy/configuration/binds?frontend=front_downstream&transaction_id=653419f0-0326-46fd-a7ce-cc528362b9bc
DEBU[0018] sending dataplane req: POST /v2/services/haproxy/configuration/log_targets?parent_type=frontend&parent_name=front_downstream&transaction_id=653419f0-0326-46fd-a7ce-cc528362b9bc
ERRO[0018] error calling POST /v2/services/haproxy/configuration/log_targets?parent_type=frontend&parent_name=front_downstream&transaction_id=653419f0-0326-46fd-a7ce-cc528362b9bc: response was 422: "{"code":602,"message":"index in body is required"}"
DEBU[0019] sending dataplane req: GET /v2/services/haproxy/stats/native

Thanks