valyala / gozstd

go wrapper for zstd

gozstd - go wrapper for zstd

Features

• Vendors upstream zstd without any modifications.

• Simple API.

• Optimized for speed. The API may be easily used in zero allocations mode.

• Compress* and Decompress* functions are optimized for high concurrency.

• Proper Writer.Flush for network apps.

• Supports the following features from upstream zstd:

  • Block / stream compression / decompression with all the supported compression levels and with dictionary support.
  • Dictionary building from a sample set. The created dictionary may be saved to persistent storage / transfered over the network.
  • Dictionary loading for compression / decompression.

  Pull requests for missing upstream zstd features are welcome.

Quick start

How to install gozstd?

go get -u github.com/valyala/gozstd

How to compress data?

The easiest way is just to use Compress:

	compressedData := Compress(nil, data)

There is also StreamCompress and Writer for stream compression.

How to decompress data?

The easiest way is just to use Decompress:

	data, err := Decompress(nil, compressedData)

There is also StreamDecompress and Reader for stream decompression.

How to cross-compile gozstd?

If you're cross-compiling some code that uses gozstd and you stumble upon the following error:

# github.com/valyala/gozstd
/go/pkg/mod/github.com/valyala/[email protected]/stream.go:31:59: undefined: CDict
/go/pkg/mod/github.com/valyala/[email protected]/stream.go:35:64: undefined: CDict
/go/pkg/mod/github.com/valyala/[email protected]/stream.go:47:20: undefined: Writer

You can easily fix it by enabling CGO and using a cross-compiler (e.g. arm-linux-gnueabi-gcc):

env CC=arm-linux-gnueabi-gcc GOOS=linux GOARCH=arm CGO_ENABLED=1 go build ./main.go 

NOTE: Check #21 for more info.

Who uses gozstd?

• VictoriaMetrics

FAQ

• Q: Which go version is supported? A: go1.10 and newer. Pull requests for older go versions are accepted.

• Q: Which platforms/architectures are supported? A: linux/amd64, linux/arm, linux/arm64, freebsd/amd64, darwin/amd64, darwin/arm64, windows/amd64. Pull requests for other platforms/architectures are accepted.

• Q: I don't trust libzstd*.a binary files from the repo or these files dont't work on my OS/ARCH. How to rebuild them? A: Just run make clean libzstd.a if your OS/ARCH is supported.

• Q: How do I specify custom build flags when recompiling libzstd*.a? A: You can specify MOREFLAGS=... variable when running make like this: MOREFLAGS=-fPIC make clean libzstd.a.

• Q: Why the repo contains libzstd*.a binary files?
  A: This simplifies package installation with go get without the need to perform additional steps for building the libzstd*.a.

gozstd's Issues

build as a c Library, but faile

os: centos 7
want to compile into a C library with CGo， but get error：
/usr/lib/golang/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/bin/ld: /home/lcsuper/kong-zstd-go/vendor/github.com/valyala/gozstd/libzstd_linux_amd64.a(zdict.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a shared object; recompile with -fPIC
/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

Help me, please

Can you decompress this file? @valyala
I really need the information contained in this compress file and its original form is .xml. I think it's zstd+dict so I trust you
Link file: https://drive.google.com/file/d/1jwlABNJbsPcSR1uUyrhsJCmpScTDeOYx/view?usp=drivesdk

dep ensure giving error

though go get for the library works fine, when I do dep ensure gives the following error. Any pointers ?

vendor/github.com/valyala/gozstd/dict.go:8:31: fatal error: dictBuilder/zdict.h: No such file or directory

go get complains about missing gcc

Running go get complains about missing gcc

root@ba0ab0d66064:/# go get -v github.com/valyala/gozstd
github.com/valyala/gozstd (download)
github.com/valyala/gozstd
# github.com/valyala/gozstd
exec: "gcc": executable file not found in $PATH

Is this expected? I thought gcc is not necessary since the repository already contains the *.a files.

Initialization fails when running go test -race on a module which includes gozstd.

Running any test or benchmark from this file with -race will cause the below error

==17204==ERROR: ThreadSanitizer failed to allocate 0x000000a61000 (10883072) bytes at 0x200dc6f9fc000 (error code: 87)
exit status 66

package crashOnLoad


import (
	"bytes"
	"compress/gzip"
	"encoding/base64"
	"io/ioutil"
	"sync"
	"testing"

	"github.com/stretchr/testify/assert"
	"github.com/valyala/gozstd"
)


func TestThisCantRunEither(t *testing.T) {
}

func BenchmarkCompressValyalaZstd(b *testing.B) {
	b.ResetTimer()
	for i := 0; i < b.N; i += 1 {
		func() {
			compressedBytes := gozstd.Compress(make([]byte, 0, len(SampleVast)), SampleVast)
			compressionRatio := float64(len(SampleVast)) / float64(len(compressedBytes))
			if compressionRatio < 3.8 {
				panic("WTF")
			}
		}()
	}
}

func BenchmarkDecompressValyalaZstd(b *testing.B) {
	compressedBytes := gozstd.Compress(make([]byte, 0, len(SampleVast)), SampleVast)
	compressionRatio := float64(len(SampleVast)) / float64(len(compressedBytes))
	if compressionRatio < 3.8 {
		panic("WTF")
	}

	b.ResetTimer()
	for i := 0; i < b.N; i += 1 {
		func() {
			decompressedBytes, err := gozstd.Decompress(make([]byte, 0, 8192), compressedBytes)
			if err != nil {
				panic("wtf")
			}
			if len(decompressedBytes) <= 0 {
				panic("wtf")
			}
		}()
	}
}

go test -benchmem -run=^$ -coverprofile=go-code-cover -bench ^TestThisCantRunEither$ crashOnLoad -race -v -memprofile memprofile.out -cpuprofile profile.out -benchtime 3s

Creating dictionaries by reference (eg: ZSTD_create[CD]Dict_byReference)

Problem

The current library wraps these two functions from zstd:

• ZSTD_createDDict
• ZSTD_createCDict

Both of these copy the input dictionary buffer, allowing the caller to release them. My issue with this is that I am storing dictionaries used for both compression and decompression in the same programs. Compressing data written to storage, and decompressing from storage.

We have a bunch of tenants, thus a bunch of dictionaries. With the current state, this means having two copies of each dictionary.

If I understand correctly, wrapping the *_byReference functions could share the same backing dictionary with both compressors and decompressors. This requires a bit more cooperation from the client program, but this can be noted in the documentation that it's an optimization path.

What are your thoughts, @valyala?

Cross compilation fails

As reported in vmihailenco/taskq#55, if I try to cross compile a project that has a indirect dependency on this package I get:

❯ env GOOS=linux GOARCH=arm CGO_ENABLED=0 go build ./cmd/api/main.go                   
# github.com/valyala/gozstd
../../go/pkg/mod/github.com/valyala/[email protected]/stream.go:31:59: undefined: CDict
../../go/pkg/mod/github.com/valyala/[email protected]/stream.go:35:64: undefined: CDict
../../go/pkg/mod/github.com/valyala/[email protected]/stream.go:47:20: undefined: Writer

Add support for custom window sizes (long distance matching)

The default for zstd appears to be 27 (the --long option). It would be great if this could be exposed via this module. I believe this is the relevant code from zstandard's code repository: https://github.com/facebook/zstd/blob/fbcae274a4752df2c386778808948e88fb89b295/lib/compress/zstd_ldm.c

Thanks for the awesome work!

Bindings for seekable format

I'm about: https://github.com/facebook/zstd/tree/dev/contrib/seekable_format

Issue building in docker

Having trouble using this library in a new docker image:

Dockerfile

FROM golang:1.10-stretch

RUN apt-get install gcc git make

RUN go get -u github.com/valyala/gozstd

Error output:

Step 1/9 : FROM golang:1.10-stretch
 ---> 6fe15d4cbc64
Step 2/9 : RUN apt-get install gcc git make
 ---> Using cache
 ---> b5f3a286c946
Step 3/9 : RUN go get -u github.com/valyala/gozstd
 ---> Running in 603c8d4e6a82
# github.com/valyala/gozstd
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(zstd_common.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(zstd_compress.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(zstd_opt.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(zstd_decompress.o): relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(zdict.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(error_private.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(fse_decompress.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(fse_compress.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(huf_decompress.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(cover.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(divsufsort.o): relocation R_X86_64_32S against `.rodata' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: src/github.com/valyala/gozstd/libzstd.a(pool.o): relocation R_X86_64_32 against `.bss' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
The command '/bin/sh -c go get -u github.com/valyala/gozstd' returned a non-zero code: 2

Build fails on Alpine due to bundled .a files

Firstly, thanks a lot for making this. I was experiencing DataDog/zstd#22 and switching over to this library seems to have solved that problem, with a minimum of code changes.

My issue is that I deploy my golang application in docker containers. I use Alpine Linux as my deployment base in order to reduce the container sizes. Because Alpine uses musl instead of glibc, the pre-compiled binaries you distribute cannot work with Alpine. Instead, at build time errors like this are generated:

39 # github.com/valyala/gozstd
40 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_analyzeEntropy':
41 zdict.c:(.text+0x7ce): undefined reference to `__fprintf_chk'
42 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_analyzePos':
43 zdict.c:(.text+0x1832): undefined reference to `__fprintf_chk'
44 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: zdict.c:(.text+0x18c2): undefined reference to `__fprintf_chk'
45 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_finalizeDictionary':
46 zdict.c:(.text+0x1b24): undefined reference to `__fprintf_chk'
47 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o): in function `ZDICT_trainFromBuffer_unsafe_legacy':
48 zdict.c:(.text+0x1d54): undefined reference to `__fprintf_chk'
49 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(zdict.o):zdict.c:(.text+0x1dd4): more undefined references to `__fprintf_chk' follow
50 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(entropy_common.o): in function `FSE_readNCount':
51 entropy_common.c:(.text+0x345): undefined reference to `__memcpy_chk'
52 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(fastcover.o): in function `FASTCOVER_buildDictionary.isra.6':
53 fastcover.c:(.text+0x3a9): undefined reference to `__fprintf_chk'
54 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: fastcover.c:(.text+0x44f): undefined reference to `__fprintf_chk'
55 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: fastcover.c:(.text+0x488): undefined reference to `__fprintf_chk'
56 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(fastcover.o): in function `FASTCOVER_ctx_init':
57 fastcover.c:(.text+0x8a0): undefined reference to `__fprintf_chk'
58 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: fastcover.c:(.text+0x8e6): undefined reference to `__fprintf_chk'
59 /usr/lib/gcc/x86_64-alpine-linux-musl/8.3.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/pkg/mod/github.com/valyala/[email protected]/libzstd_linux_amd64.a(fastcover.o):fastcover.c:(.text+0xbc7): more undefined references to `__fprintf_chk' follow
60 collect2: error: ld returned 1 exit status

I can work around this by putting in special case code into my build process just for this module, something like

go get -d github.com/valyala/gozstd; cd /go/pkg/mod/github.com/valyala/gozstd@*; if [[ ! -f _rebuilt ]]; then chmod -R +w .; make -j8 clean; make -j8 libzstd.a; touch _rebuilt; fi; cd /go/src/myproject

...but it feels quite unusual to have to do that.

What changes would be required to avoid having to distribute a binary library?

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

Hi!

In our project, we use the gozstd library to compress/decompress records. We ran into the issue where Go completely crashes the program about once in 50 million calls to decompress. Here's one stack trace:

runtime: pointer 0xc007e64a76 to unallocated span span.base()=0xc007dd8000 span.limit=0xc007ed7e00 span.state=3
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw(0x51453e9, 0x3e)
        /usr/local/go/src/runtime/panic.go:617 +0x72 fp=0x7f05f5bd5c88 sp=0x7f05f5bd5c58 pc=0x4e2e42
runtime.findObject(0xc007e64a76, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/runtime/mbitmap.go:397 +0x3bd fp=0x7f05f5bd5cd8 sp=0x7f05f5bd5c88 pc=0x4c9d5d
runtime.wbBufFlush1(0xc0000a4500)
        /usr/local/go/src/runtime/mwbbuf.go:288 +0xae fp=0x7f05f5bd5d30 sp=0x7f05f5bd5cd8 pc=0x4df32e
runtime.wbBufFlush.func1()
        /usr/local/go/src/runtime/mwbbuf.go:218 +0x3b fp=0x7f05f5bd5d48 sp=0x7f05f5bd5d30 pc=0x50e63b
runtime.systemstack(0xc000001380)
        /usr/local/go/src/runtime/asm_amd64.s:351 +0x66 fp=0x7f05f5bd5d50 sp=0x7f05f5bd5d48 pc=0x510486
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1153 fp=0x7f05f5bd5d58 sp=0x7f05f5bd5d50 pc=0x4e73f0

goroutine 547151 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc001a93020 sp=0xc001a93018 pc=0x510410
runtime.wbBufFlush(0xc00ce01290, 0xc00617d380)
        /usr/local/go/src/runtime/mwbbuf.go:206 +0x66 fp=0xc001a93050 sp=0xc001a93020 pc=0x4df1f6
runtime.gcWriteBarrier(0x4b168e0, 0xc00ce01290, 0x599a2b, 0xc00123b110, 0x0, 0x0, 0xc00123b110, 0x502028, 0x17d72e615fba, 0xc00ccdc0f3, ...)
        /usr/local/go/src/runtime/asm_amd64.s:1424 +0xb4 fp=0xc001a930d8 sp=0xc001a93050 pc=0x5125a4
bytes.NewBuffer(...)
        /usr/local/go/src/bytes/buffer.go:451

goroutine 546998 [runnable]:
runtime.cgocall(0x4089750, 0xc006645060, 0xc000096500)
        /usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc006645030 sp=0xc006644ff8 pc=0x4b826b
github.com/valyala/gozstd._Cfunc_ZSTD_initDStream_usingDDict(0x7f05e0000c80, 0x7f05d0001f80, 0x5)
        bazel-out/k8-opt/bin/go/src/github.com/valyala/gozstd/1.12/_cgo_gotypes.go:610 +0x4e fp=0xc006645060 sp=0xc006645030 pc=0x1832a7e
github.com/valyala/gozstd.initDStream.func1(0x7f05e0000c80, 0x7f05d0001f80, 0xc0066450c0)
        go/src/github.com/valyala/gozstd/reader.go:95 +0x8f fp=0xc006645098 sp=0xc006645060 pc=0x1835baf
github.com/valyala/gozstd.initDStream(0x7f05e0000c80, 0xc000384d08)
        go/src/github.com/valyala/gozstd/reader.go:95 +0x3d fp=0xc0066450c0 sp=0xc006645098 pc=0x1833d3d
github.com/valyala/gozstd.NewReaderDict(0x5736bc0, 0xc0142a2ab0, 0xc000384d08, 0xc00125c450)
        go/src/github.com/valyala/gozstd/reader.go:50 +0x3e fp=0xc006645110 sp=0xc0066450c0 pc=0x1833a2e

I got a few stack traces. The common theme with the stack traces is the decompression using the gozstd library as well as a call to runtime.gcWriteBarrier (can be in an unrelated goroutine).

I am kind of at a loss about how to debug this, happy to try something/add logs if you have suggestions.

Go version: 1.12
gozstd version: caebc4659becc90614b8d281a497fd08785b92ff
Platform: linux

checksum support?

Is it possible to support checksumming? Zstd has an entry point

size_t ZSTD_compress_advanced(ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize,
const void* dict,size_t dictSize,
ZSTD_parameters params);

that allows adding a checksum in the trailer. It would be great if gozstd supports it.

Add darwin/arm64 support

MacBook with M1 chip

Update to Zstd 1.5.0

Hi,

I see Zstd 1.5.0 has just been released, with improved performance.

https://github.com/facebook/zstd/releases/tag/v1.5.0

I tried running make update-zstd in a copy of the gozstd repo - to submit a pull request - but when I committed the changes afterwards, there was only a single file that had changed. It didn't seem to have updated. I suspect there is more to this process?

Please update gozstd to use Zstd 1.5.0 - thanks!

Cgo wrappers cannot safely pass uintptr to C functions

This package passes uintptr to C functions, which is not safe. It can cause memory corruption in rare cases. I did not try to reproduce this bug with this package, because it is subtle. However, we have a package with similar code, and we ended up with a memory corruption bug: DataDog/zstd#90 . I'm filing this issue because I noticed this package uses the same pattern, so it probably has the same problem.

The unlucky sequence is:

• Code evaluates the uintptr from a Go pointer to a variable on the stack.
• Code calls the Cgo wrapper.
• The Go runtime decides this goroutine needs a larger stack, and copies it somewhere else. It fixes the pointers to variables on the stack. It cannot fix the uintptr variable, since it does not think it is a pointer.
• Another thread uses the previous stack and changes the contents.
• The C code casts the uintptr to a pointer, then reads from the wrong memory.

One example piece of code that does this is: https://github.com/valyala/gozstd/blob/master/gozstd.go#L17 , however there are many others. In many cases, I think using unsafe.Pointer instead will not impact the performance of gozstd. For example, the Writer API already copies data into its own heap-allocated buffer, before calling Cgo.

For details, see my reproduction of this problem: https://github.com/evanj/cgouintptrbug

StreamCompress() returns different from Compress() result

go version: 1.14.2

Test:

func TestStream(t *testing.T) {
	buffer := []byte("this is test data for StreamCompress vs Compress comparison")
	reader := bytes.NewReader(buffer)
	writer := bytes.NewBuffer(nil)
	err := zstd.StreamCompress(writer, reader)
	if err != nil {
		t.Fatal(err)
	}
	streamResult := writer.Bytes()
	compressResult := zstd.Compress(nil, buffer)
	assert.Equal(t, streamResult, compressResult)
}

fails with

=== RUN   TestStream
	assert.go:24: got [40 181 47 253 0 88 216 1 0 116 104 105 115 32 105 115 32 116 101 115 116 32 100 97 116 97 32 102 111 114 32 83 116 114 101 97 109 67 111 109 112 114 101 115 115 32 118 115 32 67 111 109 112 114 101 115 115 32 99 111 109 112 97 114 105 115 111 110 1 0 0] want [40 181 47 253 32 59 217 1 0 116 104 105 115 32 105 115 32 116 101 115 116 32 100 97 116 97 32 102 111 114 32 83 116 114 101 97 109 67 111 109 112 114 101 115 115 32 118 115 32 67 111 109 112 114 101 115 115 32 99 111 109 112 97 114 105 115 111 110] 

--- FAIL: TestStream (0.00s)
FAIL

Seems like StreamCompress adds 0x01, 0x00, 0x00 bytes at the end.

panic while decompressing data

runtime: pointer 0xc028c4c517 to unused region of span span.base()=0xc00b0c0000 span.limit=0xc00b0c1e00 span.state=1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:

goroutine 64531572 [syscall]:
runtime.cgocall(0x22e74d0, 0xc02f3cb988)
/usr/local/go/src/runtime/cgocall.go:156 +0x5c fp=0xc02f3cb960 sp=0xc02f3cb928 pc=0x413a1c
github.com/valyala/gozstd._Cfunc_ZSTD_decompressStream_wrapper(0x7fdd94138050, 0x7fdd9416f760, 0x7fdd9414f730)
_cgo_gotypes.go:387 +0x4d fp=0xc02f3cb988 sp=0xc02f3cb960 pc=0xba10ed
github.com/valyala/gozstd.(*Reader).fillOutBuf(0xc02679a680)
/opt/tiger/compile_path/pkg/mod/github.com/valyala/[email protected]/reader.go:205 +0xa5 fp=0xc02f3cb9e0 sp=0xc02f3cb988 pc=0xba2945
github.com/valyala/gozstd.(*Reader).Read(0xc02679a680, {0xc04efd2000, 0x7fe00, 0x203005})
/opt/tiger/compile_path/pkg/mod/github.com/valyala/[email protected]/reader.go:179 +0x4b fp=0xc02f3cba10 sp=0xc02f3cb9e0 pc=0xba278b
bytes.(*Buffer).ReadFrom(0xc014e43cb0, {0x2c06000, 0xc02679a680})
/usr/local/go/src/bytes/buffer.go:204 +0x98 fp=0xc02f3cba68 sp=0xc02f3cba10 pc=0x504f38

byte size grow after compress

Hi @valyala

There is a case that makes me confused. Why byte size grow after compress? the example code is below.

package main

import (
	"fmt"

	"github.com/valyala/gozstd"
)

func main() {
	data := []byte{0, 0, 253, 53, 0, 1, 3, 197, 161, 198, 102, 102, 102, 106, 3, 226, 127, 42, 170, 170, 170, 239, 114, 223, 255, 255, 255, 229, 234, 92, 170, 170, 170, 170, 189, 203, 230, 102, 102, 102, 119, 153, 184, 204, 204, 204, 205, 68, 170, 170, 170, 171, 189, 77, 191, 255, 255, 255, 202, 74, 170, 170, 170, 169, 222, 159, 115, 51, 51, 51, 58, 24, 204, 204, 204, 205, 32, 149, 85, 85, 85, 114, 59, 255, 255, 255, 252, 160, 149, 85, 85, 85, 82, 31, 51, 51, 51, 51, 160, 140, 204, 204, 204, 210, 121, 85, 85, 85, 87, 32, 191, 255, 255, 255, 202, 25, 85, 85, 85, 85, 35, 115, 51, 51, 51, 63, 255, 255, 255, 255, 128}
	fmt.Println("len before compress:", len(data))
	compressedData := gozstd.Compress(nil, data)
	fmt.Println("len after compress:", len(compressedData))
}

Output:

len before compress: 131
len after compress: 140

how can I reduce the byte sizes and compress them actually？

Go modules set gozstd directory to 0555, causing "make clean libzstd.a" to fail

Issue

make clean libzstd.a fails with a file system permission error if using Go modules as implemented in Go 1.11. The make clean target tries to remove the libzstd_target file, but Go has set the current directory to Unix permission 0555, which does not allow removing files.

Specifically:

$ make clean libzstd.a
rm -f libzstd_darwin.a
rm: libzstd_darwin.a: Permission denied
make: *** [clean] Error 1

Seems that with the move to modules, the Go project is taking an aggressive stance on this. Their deciding reason is that this, in particular, prevents unit tests, which are run from their containing directory, from modifying the contents of their directory.

See golang/go#27161.

Expected

Either make clean libzstd.a succeeds or there is some workaround.

Steps to reproduce

Bash script:

export GOPATH=/tmp/go
mkdir $GOPATH
cd $GOPATH
mkdir main
cd main
cat > main.go <<EOF
package main

import "github.com/valyala/gozstd"

func main() {
    dst := make([]byte, 1024)
    src := make([]byte, 1024)
    gozstd.Compress(dst, src)
}
EOF
cat > go.mod <<EOF
module main

require github.com/valyala/gozstd v1.2.1
EOF
go get
cd $GOPATH/pkg/mod/github.com/valyala/[email protected]
make clean libzstd.a

Documentation of which zstd features are unsupported?

In some comments on social media, you mentioned that some features (block, parallel) are not supported. Would be great to have a quick blurb about which features are not supported in the README. Thanks!

Using a customized old version of c++ zstd... using this wrapper... issue

great work by the way.

i'm using a very old and customized version of zstd. how do i replace

i've compiled my own c++ zstd into /usr/local/lib but... error here:

is it possible for the c wrapper to use different library files separately to compile? how to do this?

otherwise, how do i replace the 3 functions below? coz they are in the newer zstd package

CGO_CFLAGS="-I/usr/local/include" CGO_LDFLAGS="-L/usr/local/lib -lboost_fiber -lboost_context -lstdc++ -lm -lz -lsnappy -llz4 -lzstd -lbz2 -ljemalloc -pthread -lm -lgomp -lrt -ldl -laio" go get -u github.com/[myownrepo]/frocks
 

# github.com/valyala/gozstd


/usr/bin/ld: $WORK/b049/_x003.o: in function `_cgo_e9b7c5fe0b6f_Cfunc_ZSTD_getFrameContentSize_wrapper':
gozstd.cgo2.c:(.text+0x29b): undefined reference to `ZSTD_getFrameContentSize'
/usr/bin/ld: $WORK/b049/_x005.o: in function `_cgo_e9b7c5fe0b6f_Cfunc_ZSTD_CCtx_refCDict_wrapper':
writer.cgo2.c:(.text+0x1b): undefined reference to `ZSTD_CCtx_refCDict'
/usr/bin/ld: $WORK/b049/_x005.o: in function `_cgo_e9b7c5fe0b6f_Cfunc_ZSTD_CCtx_setParameter_wrapper':
writer.cgo2.c:(.text+0x5d): undefined reference to `ZSTD_CCtx_setParameter'
collect2: error: ld returned 1 exit status

`NewWriterParams` fails when `confluent-kafka-go` is in dependencies

I got a panic in this code

package main

import (
	"bytes"

	"github.com/confluentinc/confluent-kafka-go/kafka"
	"github.com/valyala/gozstd"
)

func main() {
	var _ kafka.ConfigMap
	gozstd.NewWriterParams(&bytes.Buffer{}, &gozstd.WriterParams{})
}

Trace:

panic: BUG: unexpected error in ZSTD_CCtx_setParameter: Operation not authorized at current processing stage

goroutine 1 [running]:
github.com/valyala/gozstd.ensureNoError(0x8bf4e4, 0x16, 0xffffffffffffffc4)
        /home/iwfg/go/src/confluentkafka-zstd/vendor/github.com/valyala/gozstd/gozstd.go:332 +0x13f
github.com/valyala/gozstd.initCStream(0x1c95820, 0x0, 0x0, 0x0)
        /home/iwfg/go/src/confluentkafka-zstd/vendor/github.com/valyala/gozstd/writer.go:232 +0x96
github.com/valyala/gozstd.NewWriterParams(0x8dde18, 0xc0001001b0, 0xc000055f60, 0x0)
        /home/iwfg/go/src/confluentkafka-zstd/vendor/github.com/valyala/gozstd/writer.go:161 +0x66
main.main()
        /home/iwfg/go/src/confluentkafka-zstd/main.go:12 +0x70
exit status 2

Issue building in docker

Having trouble using this library in a new docker image:

Jenkinsfile:
sh 'go get -d github.com/valyala/gozstd'
sh 'cd ${GOPATH}/src/github.com/valyala/gozstd && MOREFLAGS=-fPIC make clean libzstd.a && go install'
sh 'cd ${GOPATH} && docker run -v ${GOPATH}:/go build:0.0.1 go install -buildmode=c-shared -ldflags "-s -w -extldflags=-fPIC" ${SRC}

Error output:
# github.com/valyala/gozstd /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_common.o): relocation R_X86_64_32 against .rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_compress.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_compress_sequences.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_compress_superblock.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_lazy.o): relocation R_X86_64_32 against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_ldm.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_opt.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_decompress.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_decompress_block.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zdict.o): relocation R_X86_64_32 against .rodata.str1.8' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(error_private.o): relocation R_X86_64_32 against .rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(fse_decompress.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_v05.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_v06.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(zstd_v07.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(fse_compress.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(huf_compress.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(huf_decompress.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(divsufsort.o): relocation R_X86_64_32S against .rodata' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(fastcover.o): relocation R_X86_64_32 against undefined symbol __iob' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(pool.o): relocation R_X86_64_32 against .bss' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: /go/src/github.com/valyala/gozstd/libzstd_illumos_amd64.a(cover.o): relocation R_X86_64_32 against .text' can not be used when making a shared object; recompile with -fPIC /usr/lib/gcc/x86_64-alpine-linux-musl/6.4.0/../../../../x86_64-alpine-linux-musl/bin/ld: final link failed: Nonrepresentable section on output collect2: error: ld returned 1 exit status script returned exit code 2

@valyala Please help on the issue

gozstd fails during linking if the standard lib "plugin" module is imported

you can reproduce this simply by importing both gozstd and plugin at the same time, for example

package main

import (
    _ "github.com/valyala/gozstd"
    _ "plugin"
)

as an example I forked the gozstd repo to add a test to fail this, thus

go get github.com/gordallott/gozstd
go test -v github.com/gordallott/gozstd

should result in something like

# github.com/gordallott/gozstd.test
/home/mhr3/.axiom-dev/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /home/mhr3/.axiom-dev/gopath/src/github.com/gordallott/gozstd/libzstd_linux_amd64.a(zdict.o): relocation R_X86_64_PC32 against symbol `stderr@@GLIBC_2.2.5' can not be used when making a PDE object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

tested on multiple machines with go version go version go1.15.2 linux/amd64

does not appear to be a CGO problem as using other CGO based modules work fine with the plugin module