cafxx / httpcompression

Go middleware to compress HTTP responses with Gzip, Deflate, Brotli, Zstandard, XZ/LZMA2, LZ4, and more..

Golang HTTP server middleware for gzip/brotli/zstandard compression

This is a small Go package which wraps HTTP handlers to transparently compress response bodies using zstd, brotli, gzip or deflate - for clients which support them. Although it's usually simpler to leave that to a reverse proxy (like nginx or Varnish), this package is useful when that is undesirable. In addition, this package allows users to extend it by plugging in third-party or custom compression encoders.

Note: This package was recently forked from the dead NYTimes/gziphandler. Maintaining drop-in compatibility is not a goal of this fork, as the scope of this fork is significantly wider than the original package.

⚠️ As we have not reached 1.0 yet, API is still subject to changes.

Features

• gzip, deflate, brotli, and zstd compression by default, alternate (faster) gzip, zstd implementations are optional
• Apply compression only if response body size is greater than a threshold
• Apply compression only to a allowlist/denylist of MIME content types
• Define encoding priority (e.g. give brotli a higher priority than gzip)
• Control whether the client or the server defines the encoder priority
• Plug in third-party/custom compression schemes or implementations
• Custom dictionary compression for zstd and deflate
• Low memory alliocations via transparent encoder reuse

Demo

While no dedicated demo exists, the demo website for regexp2go internally uses httpcompression to transparently compress responses.

Install

go get github.com/CAFxX/httpcompression

Usage

Call httpcompression.DefaultAdapter to get an adapter that can be used to wrap any handler (an object which implements the http.Handler interface), to transparently provide response body compression. Note that httpcompression automatically compresses using Zstandard, Brotli, Deflate, and Gzip depending on the capabilities of the client (Accept-Encoding) and the configuration of this handler (by default, Zstandard, Brotli and gzip are all enabled and, conditional on client support, used in that order of preference).

As a simple example:

package main

import (
    "io"
    "net/http"
    "github.com/CAFxX/httpcompression"
)

func main() {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain")
        io.WriteString(w, "Hello, World")
    })
    compress, _ := httpcompression.DefaultAdapter() // Use the default configuration
    http.Handle("/", compress(handler))
    http.ListenAndServe("0.0.0.0:8080", nil)
}

Pluggable compressors

It is possible to use custom compressor implementations by specifying a CompressorProvider for each of the encodings the adapter should support. This also allows to support arbitrary Content-Encoding schemes (e.g. lzma, or zstd with a static dictionary - see the examples).

pgz, err := httpcompression.Compressor("gzip", 0, pgzip.New(pgzip.Options{Level: 6}))
if err != nil {
    log.Fatal(err)
}
compress, err := httpcompression.Adapter(
    // use klauspost/pgzip as compressor for the "gzip" content-encoding, with priority 0
    pgz,
)
if err != nil {
    log.Fatal(err)
}
http.Handle("/", compress(handler))

The contrib/ directory contains a number of bundled implementations that are ready for use:

Content-Encoding	Provider package	Implementation package	Notes	Dictionary	Go/cgo	Default	IANA registry
deflate	contrib/compress/zlib	compress/zlib	Slower than klauspost/zlib	Yes	Go	Yes	Yes
deflate	contrib/klauspost/zlib	github.com/klauspost/compress/zlib		Yes	Go	No	Yes
gzip	contrib/compress/gzip	compress/gzip	Slower than klauspost/gzip	No	Go	Yes	Yes
gzip	contrib/klauspost/gzip	github.com/klauspost/compress/gzip		No	Go	No	Yes
gzip	contrib/klauspost/pgzip	github.com/klauspost/pgzip	Parallel compression	No	Go	No	Yes
zstd	contrib/klauspost/zstd	github.com/klauspost/compress/zstd		Yes	Go	Yes	Yes
zstd	contrib/valyala/gozstd	github.com/valyala/gozstd	Slower than klauspost/zstd	Yes	cgo	No	Yes
brotli	contrib/andybalholm/brotli	github.com/andybalholm/brotli	Slower than google/brotli	No	Go	Yes	Yes
brotli	contrib/google/cbrotli	github.com/google/brotli	Requires brotli libraries to be installed	No	cgo	No	Yes
lz4	contrib/pierrec/lz4	github.com/pierrec/lz4/v4		No	Go	No	No
xz	contrib/ulikunitz/xz	github.com/ulikunitz/xz		No	Go	No	No

Framework integration

In addition to the default support for net/http, httpcompression provides adapters for the following web frameworks:

Framework	Adapter
github.com/gofiber/fiber/v2	contrib/gofiber/fiber/v2
github.com/labstack/echo	contrib/labstack/echo
github.com/gin-gonic/gin	contrib/gin-gonic/gin

Benchmark

See the benchmark results to get an idea of the relative performance and compression efficiency of gzip, brotli and zstd in the current implementation.

TODO

• Add dictionary support to brotli (zstd and deflate already support it, gzip does not allow dictionaries)
• Allow to choose dictionary based on content-type
• Provide additional implementations based on the bindings to the original native implementations
• Add compressed payload caching (if the same payload has already been compressed and is present in the cache, skip compression)
• Add write buffering (compress larger chunks at once)
• Add decompression (if the payload is already compressed but the client supports better algorithms, or does not support a certain algorithm)
• Add other, non-standardized content encodings (lzma/lzma2/xz, snappy, bzip2, etc.)
• Dynamically tune MinSize (and possibly also ContentTypes, level/quality, ...)
• Automatically generate and serve dictionaries

License

Apache 2.0.

httpcompression's Issues

panic when echo HTTPErrorHandler is triggered

package main

import (
	"github.com/CAFxX/httpcompression"
	"github.com/labstack/echo/v4"
)

func main()  {
	e:=echo.New()
	adapter, err := httpcompression.DefaultAdapter()
	if err != nil {
		return
	}
	e.Use(echo.WrapMiddleware(adapter))
	e.Start(":1323")
}

Use the above code, and then access an unregistered path to trigger HTTPErrorHandler and report the following error:

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.11.4
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:1323
echo: http: panic serving 127.0.0.1:64345: runtime error: invalid memory address or nil pointer dereference
goroutine 10 [running]:
net/http.(*conn).serve.func1()
	xxx/sdk/go1.21.0/src/net/http/server.go:1868 +0xb9
panic({0x11aa500?, 0x14c6d50?})
	xxx/sdk/go1.21.0/src/runtime/panic.go:920 +0x270
github.com/CAFxX/httpcompression.(*compressWriter).Header(0xc00008f920?)
	<autogenerated>:1 +0x1e
github.com/labstack/echo/v4.(*Response).Header(...)
	xxx/go/pkg/mod/github.com/labstack/echo/[email protected]/response.go:36
github.com/labstack/echo/v4.(*context).writeContentType(0x10321b9?, {0x12069cd, 0x1f})
	xxx/go/pkg/mod/github.com/labstack/echo/[email protected]/context.go:229 +0x30
github.com/labstack/echo/v4.(*context).json(0xc0000600a0, 0x194, {0x11abba0, 0xc00001ec30}, {0x0, 0x0})
	xxx/go/pkg/mod/github.com/labstack/echo/[email protected]/context.go:492 +0x45
github.com/labstack/echo/v4.(*context).JSON(0xc0000600a0, 0xc00001ec30?, {0x11abba0, 0xc00001ec30})
	xxx/go/pkg/mod/github.com/labstack/echo/[email protected]/context.go:502 +0xc7
github.com/labstack/echo/v4.(*Echo).DefaultHTTPErrorHandler(0xc000134240, {0x1281940, 0xc00001ed80?}, {0x1287648, 0xc0000600a0})
	xxx/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:460 +0x46c
github.com/labstack/echo/v4.(*Echo).ServeHTTP(0xc000134240, {0x1283820?, 0xc0000942a0}, 0xc000518100)
	xxx/go/pkg/mod/github.com/labstack/echo/[email protected]/echo.go:670 +0x3bb
net/http.serverHandler.ServeHTTP({0xc00001ea50?}, {0x1283820?, 0xc0000942a0?}, 0x6?)
	xxx/sdk/go1.21.0/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc00012c120, {0x1283d50, 0xc00051a2d0})
	xxx/sdk/go1.21.0/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 1
	xxx/sdk/go1.21.0/src/net/http/server.go:3086 +0x5cb

zstd compression level not possible?

In

httpcompression/adapter.go

Lines 210 to 216 in 102a9fb

	func BrotliCompressionLevel(level int) Option {
	c, err := brotli.New(brotli.Options{Quality: level})
	if err != nil {
	return errorOption(err)
	}
	return BrotliCompressor(c)
	}

gzip, br, deflate compression levels are configurable via option but I cannot see zstd level? It would be nice to have.

Feature request: add hooks to enable compression metrics

Hi there. First off, thank you for the high-quality library. I rolled my own fork of the https://github.com/nytimes/gziphandler and was contemplating a brotli version before happily discovering your repo.

I recently did an optimization pass on my server. I noticed the server was spending 200 ms to encode large RPC responses using the Brotli default compression level of 6. It took a while to track down because I had to infer the time spent on compression by adding logs to the handler before and after the httpcompression handler. I figured other folks would benefit from metrics, hence this feature request.

I'd like to be able to emit traces and metrics for compression to answer the following questions:

• How long did the compression take?
• Which compression algorithm was used, and at what compression level?
• What was the compression ratio?
• What was the input size and output size?

I've seen a few approaches to support tracing.

• Go standard library HTTP tracing by embedding a special struct in the context: https://go.dev/blog/http-tracing.
• PGX uses a QueryTracer interface added to the ConnConfig struct.

The struct is advantageous because you can add new methods without breaking existing clients. With interfaces, the library author needs to add new interfaces for additional functionality and check to see if the user-provided tracer supports the new interface, i.e., interface smuggling.

I like the httptrace approach of using a single struct with optional methods. As a straw man, maybe something like:

func handleRequest() {
	compressHandler, err := httpcompression.Adapter(
		httpcompression.BrotliCompressionLevel(brotli.BestSpeed),
		// Need a factory here to create a new trace for each request.
		httpcompression.TraceProvider(func(ctx context.Context) *CompressTrace {
			span := trace.SpanFromContext(ctx)
			return &CompressTrace{
				CompressStart: func(info CompressStartInfo) {
					span.AddEvent("compress start",
						trace.String("content_type", info.ContentType),
						trace.String("content_encoding", info.ContentEncoding),
					)
				},
				CompressDone: func(info CompressDoneInfo) {
					span.AddEvent("compress done",
						trace.Int("bytes_read", info.BytesRead),
						trace.Int("bytes_written", info.BytesWritten),
						trace.Float64("compression_ratio", float64(info.BytesWritten)/float64(info.BytesRead)),
					)
				},
			}
		}),
	)
}

// NOTE: structs are a bit overkill compared to inlining the args into the function call.
// The benefit is the library can add new fields without breaking existing clients.
// httptrace uses a mixture of structs and inlined args.
type CompressStartInfo struct {
	// ContentType is the content type of the response.
	ContentType string
	// ContentEncoding is the encoding used for the response.
	ContentEncoding string
}

type CompressDoneInfo struct {
	BytesRead    int
	BytesWritten int
	// Others:
	// * ContentEncoding - if it can change from CompressStartInfo
	// * allocations - not sure if feasible
}

type CompressTrace struct {
	// StartCompress is called when compression starts.
	CompressStart func(info CompressStartInfo)
	CompressDone  func(info CompressDoneInfo)
	// Others:
	// * WroteHeaders
	// * Close
}

Buffer data before sending it to the compressors

Compressors generally operate faster if they are provided with large chunks of data.

There is already a small buffer, but it's only used for content-type sniffing before compression start (and is bypassed entirely once compression has started).

We could instead replace it with a buffer of configurable size (by default larger than the current content-type sniffing one) that would be used also during compression to compress data in larger chunks.

Some compressors may do the same internally, so we would need to avoid double buffering in this case.

Such a buffer could later also be used to capture the full uncompressed payload of (small) responses, that paired together with the resulting compressed payload could be cached to avoid having to recompress the same data over and over.

Never close the original responsewriter

Issue using server side events support

When I use SSEs without compression, I get each message flushed back one at a time. When I enable the compression it seems to continue to buffer instead of flushing the data back.

Here is a small example to replicate:

curl -N http://localhost:8090
curl -N --output --compressed http://localhost:8090

package main

import (
	"fmt"
	"time"

	"github.com/CAFxX/httpcompression"
	"github.com/gin-gonic/gin"
	wraphh "github.com/turtlemonvh/gin-wraphh"
)

func main() {
	r := gin.Default()

	compress, err := httpcompression.DefaultAdapter()
	if err != nil {
		panic(err)
	}

	r.Use(wraphh.WrapHH(compress))

	r.GET("/", func(c *gin.Context) {
		// Set the appropriate headers for SSE
		c.Writer.Header().Set("Content-Type", "text/event-stream")
		c.Writer.Header().Set("Cache-Control", "no-cache")
		c.Writer.Header().Set("Connection", "keep-alive")

		// Create a channel to send updates to the client
		updateCh := make(chan string)
		defer close(updateCh)

		// Start a goroutine to send updates
		go sendUpdates(c.Writer, updateCh)

		// Simulate sending updates every second
		for {
			updateCh <- "Hello, World! " + time.Now().Format("2006-01-02 15:04:05")
			time.Sleep(1 * time.Second)
		}
	})

	r.Run("0.0.0.0:8090")
}

func sendUpdates(w gin.ResponseWriter, updateCh <-chan string) {
	for update := range updateCh {
		// Write the update to the client
		fmt.Fprintf(w, "data: %s\n\n", update)
		w.Flush()
	}
}

Blacklisted content-types should not have Vary: Accept-Encoding

Blacklisted content-types should not have Vary: Accept-Encoding

Fix upstream issue nytimes#83

nytimes#83