GithubHelp home page GithubHelp logo

4kills / go-libdeflate Goto Github PK

View Code? Open in Web Editor NEW
25.0 2.0 7.0 211 KB

Go wrapper for the libdeflate library, supporting super fast zlib, gzip, and deflate compression, using cgo. Use case: in-memory, whole-buffered data.

License: MIT License

Go 99.82% C 0.18%
zlib zlib-port library go libdeflate wrapper gzip gzip-compression golang deflate

go-libdeflate's People

Contributors

4kills avatar haveachin avatar oflebbe avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar

Forkers

haveachin rohit-siddhw1 edaniels fufuok krejzollekk hoozecn oflebbe

go-libdeflate's Issues

deadly data example 

package main

import (
	"bytes"
	"compress/zlib"
	"encoding/hex"
	"fmt"
	"log"

	"github.com/4kills/go-libdeflate/v2"
	czlib "github.com/4kills/go-zlib"
)

var (
	s = "789c7d90316f83301085f7fc0ac45cac3b1b63cc0685542c55a4264b1784825bd1024686284851fe7b8104c454c" +
		"b8bdff7de9def6e3b6b3cf6db298dedc0b26f02384014858efb2af78ecb1573c224e24e92301e4919d158c0dd7e7" +
		"9a40a5de765f39ed76aca2e6a364caf5a17cacdbe2a7dcd8c6ab5e9d790eafab49d2ce87ac4f308724a5c5c70553" +
		"6bfe1b93f968fa214287570bcfc882c6034a0fc736bfde873f38f99aee666fccfb3ad0f043d4604122917dc1abdc" +
		"c6178deaa81a8416d60afcfba9ae8293e2cf2a5536609753f04194881be106c1de6d2eec7058c9c3180cdfc87691" +
		"f814505305c77796d9e66042e372dd2622ec001c143ff09bec7bea151f90c05f3e916cc22f57cc976f73fc74374"
	comp, _ = hex.DecodeString(s)
)

func main() {
	deWithZlib()       // nice, return error
	deWithLibdeflate() // hang, endless loop
	deWithCgoZlib()    // hang, endless loop
}

func deWithLibdeflate() {
	dc, _ := libdeflate.NewDecompressor()
	_, decompressed, err := dc.Decompress(comp, nil, libdeflate.ModeZlib)
	if err != nil {
		log.Println("deWithLibdeflate", err)
		return
	}
	fmt.Println(decompressed)
	dc.Close()
}

func deWithCgoZlib() {
	dc, _ := czlib.NewReader(nil)
	_, decompressed, err := dc.ReadBuffer(comp, nil)
	if err != nil {
		log.Println("deWithCgoZlib", err)
		return
	}
	fmt.Println(decompressed)
	_ = dc.Close()
}

func deWithZlib() {
	var decompressed bytes.Buffer
	dc, _ := zlib.NewReader(bytes.NewReader(comp))
	_, err := decompressed.ReadFrom(dc)
	if err != nil {
		log.Println("deWithZlib", err) // unexpected EOF
		return
	}
	fmt.Println(decompressed)
	_ = dc.Close()
}

Support for large buffers (>2G/4G)

Hi there,

Thank you for making this package, libdeflate is very fast. Have you tried to decompress buffers that are larger than 4G ?

Some functions work with ints, so I wonder how things will work if the decompressed buffer is larger than what an int can hold.

// DecompressGzip interfaces with c libdeflate for gzip decompression
func DecompressGzip(dc *C.decomp, inAddr, outAddr *byte, inSize, outSize int, sPtr uintptr) error {
    return parseResult(C.res(C.libdeflate_gzip_decompress(dc,
        unsafe.Pointer(inAddr), intToInt64(inSize),
        unsafe.Pointer(outAddr), intToInt64(outSize),
        C.mkPtr(C.size_t(sPtr)),
    )))
}

go.sum is not synced with your go.mod

When trying to use the latest version with go modules, I encountered a problem with your go modules:

github.com/4kills/go-libdeflate: github.com/4kills/[email protected]: parsing go.mod:
        module declares its path as: github.com/4kills/libdeflate
                but was required as: github.com/4kills/go-libdeflate

Try using go mod tidy to resolve it.

Fatal error: unknown pc caller

Panic occurs after about 10 minutes of running tool

runtime: g 1978: unexpected return pc for github.com/valyala/fasthttp.(*HostClient).Do called from 0x2c5d7
stack: frame={sp:0xc004605d18, fp:0xc004605d78} stack=[0xc004604000,0xc004606000)
0x000000c004605c18:  0x000000c00270fc80  0x000000000041023f <runtime.mallocgc+0x000000000000063f>
0x000000c004605c28:  0x0000000000000000  0x0000000000000000
0x000000c004605c38:  0x000000c0049ff500  0x000000c0018e8a00
0x000000c004605c48:  0x000000c001d23340  0x000000c001f47980
0x000000c004605c58:  0x00007fccdafb8d28  0x0000000000000040
0x000000c004605c68:  0x000000c000880400  0x000000c00a79cf80
0x000000c004605c78:  0x000000c00118fa00  0x000000c00270fca8
0x000000c004605c88:  0xc116af33d41fb7c8  0x0000005c362483be
0x000000c004605c98:  0x000000000095b120  0x000000c004605d08
0x000000c004605ca8:  0x00000000006465ba <github.com/valyala/fasthttp.(*HostClient).do+0x00000000000000ba>  0x000000c0015111e0
0x000000c004605cb8:  0x000000c001513880  0x000000c00151e380
0x000000c004605cc8:  0x000000000070d8d0  0x0000000000000006
0x000000c004605cd8:  0x000000000000003f  0x0000000000000000
0x000000c004605ce8:  0x0000000000000000  0x0000000000000001
0x000000c004605cf8:  0x000000c00a79cf80  0x0000000000000000
0x000000c004605d08:  0x000000c004605d68  0x0000000000646311 <github.com/valyala/fasthttp.(*HostClient).Do+0x0000000000000091>
0x000000c004605d18: <0x000000c00270fd68  0x0000000000451ed0 <runtime.concatstring2+0x0000000000000050>
0x000000c004605d28:  0x000000c00118fb48  0x0000000000000005
0x000000c004605d38:  0x0000000000000002  0x0000000000000000
0x000000c004605d48:  0x0000000000000000  0x000000000070d948
0x000000c004605d58:  0x000000000000003f  0x000000c00151e380
0x000000c004605d68:  0x000000c004605fb8 !0x000000000002c5d7
0x000000c004605d78: >0x0000000000004aba  0x000000c001513880
0x000000c004605d88:  0x00000000006df950  0x000000c0002ebdd0
0x000000c004605d98:  0x0000000000000000  0x0000000000000000
0x000000c004605da8:  0x0000000000000000  0x0000000000000002
0x000000c004605db8:  0x0000000000000000  0x0000000000000000
0x000000c004605dc8:  0x0000000000000000  0x0000000000000006
0x000000c004605dd8:  0x0000000000000000  0x0000000000000041
0x000000c004605de8:  0x0000000000000000  0x000000000000003f
0x000000c004605df8:  0x0000000000000050  0x0000000000000000
0x000000c004605e08:  0x0000000000000000  0x0000000000000000
0x000000c004605e18:  0x0000000000000000  0x0000000000000000
0x000000c004605e28:  0x0000000000000000  0x000000c00151e380
0x000000c004605e38:  0x000000c0015111e0  0x000000c0002ebdd0
0x000000c004605e48:  0x0000000000000000  0x000000c0013c00a0
0x000000c004605e58:  0x000000c001513880  0x0000000000000000
0x000000c004605e68:  0x0000000000000000  0x000000c0013c00f0
fatal error: unknown caller pc

Stack:

untime stack:
runtime.throw({0x6e16d4?, 0x8d0220?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7fcc91ffa770 sp=0x7fcc91ffa740 pc=0x436d7d
runtime.gentraceback(0x10?, 0x751070?, 0x0?, 0xc00118fa00, 0x0, 0x0, 0x7fffffff, 0x7fcc91ffaca8, 0x7fcc9062ce18?, 0x0)
        /usr/local/go/src/runtime/traceback.go:270 +0x1bb0 fp=0x7fcc91ffaac8 sp=0x7fcc91ffa770 pc=0x45b4f0
runtime.scanstack(0xc00118fa00, 0xc000029238)
        /usr/local/go/src/runtime/mgcmark.go:804 +0x1f2 fp=0x7fcc91ffacd0 sp=0x7fcc91ffaac8 pc=0x4204f2
runtime.markroot.func1()
        /usr/local/go/src/runtime/mgcmark.go:239 +0xb5 fp=0x7fcc91ffad20 sp=0x7fcc91ffacd0 pc=0x41f2f5
runtime.markroot(0xc000029238, 0x794, 0x1)
        /usr/local/go/src/runtime/mgcmark.go:213 +0x1a5 fp=0x7fcc91ffadc0 sp=0x7fcc91ffad20 pc=0x41efa5
runtime.gcDrain(0xc000029238, 0x3)
        /usr/local/go/src/runtime/mgcmark.go:1069 +0x39f fp=0x7fcc91ffae20 sp=0x7fcc91ffadc0 pc=0x42109f
runtime.gcBgMarkWorker.func2()
        /usr/local/go/src/runtime/mgc.go:1348 +0xad fp=0x7fcc91ffae70 sp=0x7fcc91ffae20 pc=0x41d5ad
runtime.systemstack()
        /usr/local/go/src/runtime/asm_amd64.s:496 +0x49 fp=0x7fcc91ffae78 sp=0x7fcc91ffae70 pc=0x465969

Idea: Using sync.Pool for libdeflate.Compress (or Decompress)

Hey, first of all, great library, incredibly fast!

In my use case, I needed to compress relatively small chunks of data from many go routines. Obviously one doesn't want to create a new Compressor for each such chunk of data. Usually sync.Pool would be great to pool the Compressors, the only issue is that a free-list-based structure wouldn't be able to call Close to deallocate the data when the sync.Pool stuff gets periodically garbage collected and there would be leaks.

But I came across a quite nice solution to that problem (actually, credit where credit is due, I got the idea from ChatGPT) - using runtime.SetFinalizer to register a hook executed before the Compressor is garbage collected that will call Close. It works pretty great in my benchmarks - very fast and I see no leaks. So I mostly just wanted to share the idea with anyone else dealing with a similar problem.

In addition, I think you could consider adding a pooled version of libdeflate.Compress and libdeflate.Decompress using this trick. There would be an issue supporting multiple compression levels - might need a pool for each level, alternatively only the default level would have this pooled version. Here is an example of how this could be done:

var compressorPool = sync.Pool{
	New: func() interface{} {
		compressor, err := libdeflate.NewCompressor()
		if err != nil {
			panic(err) // Can only happen if the program ran out of memory - a panic is appropriate
		}
		compressorPtr := &compressor // Pointer needed for SetFinalizer, also helps avoid interface value allocations
		runtime.SetFinalizer(compressorPtr, func(finalized *libdeflate.Compressor) {
			finalized.Close()
		})
		return &compressor
	},
}

func CompressPooled(in, out []byte, m libdeflate.Mode) (int, []byte, error) {
	compressor := compressorPool.Get().(*libdeflate.Compressor)
	defer compressorPool.Put(compressor)
	return compressor.Compress(in, out, m)
}

Do you see any issues with this approach?

I even thought about the option of registering the finalizer for all Compressors or Decompressors created by the library, thus outright negating the need to call Close (could become a deprecated noop without breaking the API), but perhaps that's going a bit too far - especially since Go may take some time before finalizers are called and the C-allocated memory will linger for a bit.

Upgrade to libdeflate 1.18

Please stay up to date with the latest version of libdeflate so that people don't run into issues that have already been fixed. Thanks!

go mod vendor support

I tried to vendor my dependencies but it failed because the libdeflate darwin library is not installed / copied over. Not sure how to fix that.

# github.com/4kills/go-libdeflate/native
clang: error: no such file or directory: '/xxx/vendor/github.com/4kills/go-libdeflate/native/libs/libdeflate_darwin_amd64.a'
FAIL	xxx/yyy [build failed]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.