cyberdelia / lzo Goto Github PK
View Code? Open in Web Editor NEWlzo implements reading and writing of lzop format compressed files
Home Page: https://pkg.go.dev/github.com/cyberdelia/lzo
License: MIT License
lzo's People
Stargazers
Watchers
lzo's Issues
Invalid header in this library
Getting this lzo: invalid header error in the library parsing the same with the java library org.anarres.lzo java an it decompress correctly
lzop: block size too small -- recompile lzop
If you are getting this error when trying to lzop -f files from this library.
This is related to a chunking issue or block size problem when writing a larger file. I've seen this error reported below
lzop -d broken.lzo
lzop: broken.lzo: block size too small -- recompile lzop
I've also seen this error cause
lzop file corrupted
The following test code will produce a working and a broken file.
go test -test.run TestLZO
import (
"bytes"
"io"
"io/ioutil"
"log"
"os"
"testing"
"github.com/cyberdelia/lzo"
)
func writeTestData() {
var testData []byte
for i := 0; i < 1000000; i++ {
testData = append(testData, []byte("a")...)
}
err := ioutil.WriteFile("./testdata", testData, 0666)
if err != nil {
log.Println(err)
}
}
func TestLZO(t *testing.T) {
lzoWorking(t)
lzoBroken(t)
}
func lzoWorking(t *testing.T) {
defer os.RemoveAll("./testdata")
writeTestData()
file, err := os.Open("./testdata")
defer file.Close()
if err != nil {
log.Println(err)
}
workingData, err := working(file)
if err != nil {
log.Println(err)
}
err = ioutil.WriteFile("./testdata.working.lzo", workingData, 0666)
if err != nil {
log.Println(err)
}
}
func lzoBroken(t *testing.T) {
defer os.RemoveAll("./testdata")
writeTestData()
file, err := os.Open("./testdata")
defer file.Close()
if err != nil {
log.Println(err)
}
data, err := ioutil.ReadAll(file)
if err != nil {
log.Println(err)
}
brokenData, err := broken(data)
if err != nil {
log.Println(err)
}
err = ioutil.WriteFile("./testdata.broken.lzo", brokenData, 0666)
if err != nil {
log.Println(err)
}
}
func working(r io.Reader) ([]byte, error) {
buff := bytes.NewBuffer(nil)
w, err := lzo.NewWriterLevel(buff, lzo.BestCompression)
if err != nil {
return nil, err
}
if _, err := io.Copy(w, r); err != nil {
return nil, err
}
if err := w.Close(); err != nil {
return nil, err
}
return buff.Bytes(), nil
}
func broken(data []byte) ([]byte, error) {
buff := bytes.NewBuffer(nil)
w, err := lzo.NewWriterLevel(buff, lzo.BestCompression)
if err != nil {
return nil, err
}
if _, err := w.Write(data); err != nil {
return nil, err
}
if err := w.Close(); err != nil {
return nil, err
}
return buff.Bytes(), nil
}
This is because when you call w.Write(data) with the full length of the data, it doesn't do any chunking. It just writes it all in one go which seems to cause the issue. This is not how this normally happens when reading from a io.Reader of type os.File. The reads from os.File are chunked, which cause this issue to disappear. This is obviously a problem if you are dealing with large byte slices that aren't os.File related and/or come from some other place.
So, as a "fix" to this problem outside of this libraries code you can always make sure the data is chunked itself going into w.Write.
// Compress does lzo best compression
func Compress(data []byte) ([]byte, error) {
buff := bytes.NewBuffer(nil)
w, err := lzo.NewWriterLevel(buff, lzo.BestCompression)
if err != nil {
return nil, err
}
// if someone passes in no data, it's safe for the library to write it
// which will write an empty lzo file
if len(data) == 0 {
if _, err := w.Write(data); err != nil {
return nil, err
}
} else {
// this is the block size that os.File uses to read data
// this block size works, don't change it.
blockSize := 32768
for i := 0; i < len(data); i += blockSize {
end := i + blockSize
if end > len(data) {
end = len(data)
}
if _, err := w.Write(data[i:end]); err != nil {
return nil, err
}
}
}
if err := w.Close(); err != nil {
return nil, err
}
return buff.Bytes(), nil
}
Find out why lzop complain
lzop complains when uncompressing a file generated why this library, even if the file is properly decompressed anyway, find out why and fix it:
lzop: unexpected end of file: pg135.txt.lzo
Return better error messages
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.