savetherbtz / zstd-seekable-format-go

Seekable ZSTD compression format implemented in Golang.

ZSTD seekable compression format implementation in Go

Seekable ZSTD compression format implemented in Golang.

This library provides a random access reader (using uncompressed file offsets) for ZSTD-compressed streams. This can be used for creating transparent compression layers. Coupled with Content Defined Chunking (CDC) it can also be used as a robust de-duplication layer.

Installation

go get -u github.com/SaveTheRbtz/zstd-seekable-format-go

Using the seekable format

Writing is done through the Writer interface:

import (
	"github.com/klauspost/compress/zstd"
	seekable "github.com/SaveTheRbtz/zstd-seekable-format-go"
)

enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.SpeedFastest))
if err != nil {
	log.Fatal(err)
}
defer enc.Close()

w, err := seekable.NewWriter(f, enc)
if err != nil {
	log.Fatal(err)
}

// Write data in chunks.
for _, b := range [][]byte{[]byte("Hello"), []byte(" "), []byte("World!")} {
	_, err = w.Write(b)
	if err != nil {
		log.Fatal(err)
	}
}

// Close and flush seek table.
err = w.Close()
if err != nil {
	log.Fatal(err)
}

NB! Do not forget to call Close since it is responsible for flushing the seek table.

Reading can either be done through ReaderAt interface:

dec, err := zstd.NewReader(nil)
if err != nil {
	log.Fatal(err)
}
defer dec.Close()

r, err := seekable.NewReader(f, dec)
if err != nil {
	log.Fatal(err)
}
defer r.Close()

ello := make([]byte, 4)
// ReaderAt
r.ReadAt(ello, 1)
if !bytes.Equal(ello, []byte("ello")) {
	log.Fatalf("%+v != ello", ello)
}

Or through the ReadSeeker:

world := make([]byte, 5)
// Seeker
r.Seek(-6, io.SeekEnd)
// Reader
r.Read(world)
if !bytes.Equal(world, []byte("World")) {
	log.Fatalf("%+v != World", world)
}

Seekable format utilizes ZSTD skippable frames so it is a valid ZSTD stream:

// Standard ZSTD Reader
f.Seek(0, io.SeekStart)
dec, err := zstd.NewReader(f)
if err != nil {
	log.Fatal(err)
}
defer dec.Close()

all, err := io.ReadAll(dec)
if err != nil {
	log.Fatal(err)
}
if !bytes.Equal(all, []byte("Hello World!")) {
	log.Fatalf("%+v != Hello World!", all)
}

zstd-seekable-format-go's Issues

progress bar when compressing

I am using the CLI to compress my files. With a rather large file, I was thinking a progress bar would be nice. Would you allow me to add one as a --progress flag?

Cryptographic hash function draft support.

Look into supporting cryptographic hash function draft: facebook/zstd#2737

While here, decoder should be future-proofed by verifying that reserved bits are set to 0.

getting size of compressed contents

It is my understanding that to get the size of the compressed contents, a Seek(0, io.SeekEnd) has to be performed. When this operation is performed, all the entries for the decompressed size appear to have to be sought.

I was trying to use a ReaderAt for HTTP Ranges, and when I did the Seek, the HTTP went denial-of-service because of how many HTTP requests it made.

I wondered if there was a way to implement a skippable frame that would include the overall decompressed size. This way, a single read could be made for the metadata.