Veeam 12 is sending Content-Length=0 in a GET request about traefik

[mmatur]

Hello @GRADeS-Gwen,

Thanks for your report, after some analyses it seems the issue is related to Go.

As you can see in the following method:

func (t *transferWriter) shouldSendContentLength() bool {
	if chunked(t.TransferEncoding) {
		return false
	}
	if t.ContentLength > 0 {
		return true
	}
	if t.ContentLength < 0 {
		return false
	}
	// Many servers expect a Content-Length for these methods
	if t.Method == "POST" || t.Method == "PUT" || t.Method == "PATCH" {
		return true
	}
	if t.ContentLength == 0 && isIdentity(t.TransferEncoding) {
		if t.Method == "GET" || t.Method == "HEAD" {
			return false
		}
		return true
	}

	return false
}

Go does not send Content-Length if the value is equal to 0.

I'm currently reading the RFC 2616 to understand if it is expected or not.

My 2 cents on that, it is a bit strange that Veeam send a GET request without body and a Content-Length header equal to 0. I will investigate and will come back to you once I have an answer.

[GRADeS-Gwen]

@mmatur Thank you for the analyse. Reading the Go method, it is clear that my case fall in the last return.
I also tried to read and understand the RFC2616, but that was not clear. So I search a bit everywhere and mostly the result is that "it is optionnal" ...

We are on the same page that sending a GET with a Content-Length header equal to 0 is strange. Furthermore, the problem did not exist in the previous version of Veeam (11), only in the newest (12.1). We have exchanged a lot of ticket and did a lot of test with there support. In the end they say that they are not able to confirm that the "Content-Length" was present in the "SignHeader" on version 11 (what the ..., not able to confirm your code ??!!) and that it can not be removed in the version 12 ...
The solution they gave us was to remove Traefik and talk directlly to the service, or use another reverse to offload the SSL ... I am not particulary a fan to that, but I think we will need to do something like that in the short term.

[mmatur]

@GRADeS-Gwen thanks for your answer.

Strange that they can not confirm their code. It could be linked to an upgrade to the aws-sdk between Veeam 11 and Veeam 12.

As you can see in the following screenshot, this code is present in go since many years, we will not be able to do anything on our side for this subject.

If I understood correctly your are using Traefik to handle TLS termination and add a ipAllowList Middleware.
One possibility is to use a TCP Router instead of HTTP Router.

tcp:
  middlewares:
    internalOnly:
      ipAllowList:
        sourceRange:
          - "127.0.0.1/32"
          - "172.16.0.0/12"
          - "192.168.10.0/24"
  routers:
    router-http-s3:
      rule: HostSNI(`s3.xxxxxxx.fr`)
      entryPoints:
        - websecure
      middlewares:
        - internalOnly
      service: service-s3
      tls: {}

  services:
    service-s3:
      loadBalancer:
        servers:
          - url: http://127.0.0.1:8081/
        passHostHeader: true

Switching from HTTP to TCP router will allow you to do the TLS termination on Traefik and forward the TCP request to your s3.

[GRADeS-Gwen]

Strange that they can not confirm their code.

In my opinion, they just dont want to search .. I found an old Veeam test server in Veeam 10, and I can confirm that the "Content-Length" was not present in the "SignHeader". I will try to upgrade to 11 and 12.0 to confirm our supposition and send them to Veeam.
I will report here if we got aything coming back from Veeam

One possibility is to use a TCP Router instead of HTTP Router.

Good call ! I did not even think about it. I transposed our config like bellow and it worked \o/. It give us less parameters, but at least it work with a bit of security.

tcp:
  routers:
    router-https-s3:
      rule: HostSNI(`s3.XXX.fr`)
      entryPoints:
        - web-secure
      middlewares:
        - internalOnly
      service: service-s3
      tls:
        certResolver: le-gandi

  services:
    service-s3:
      loadBalancer:
        servers:
          - address: "127.0.0.1:8081"

[mmatur]

Great @GRADeS-Gwen

Thanks for the feedback

I'm closing the issue because the issue is not come from Traefik but from Veeam.