madler / pigz Goto Github PK

Hi thanks for pigz it's wonderful gzip replacement from my tests http://vbtechsupport.com/1614/. But recently, I got a new server a Xeon E3-1270 which is 4 cpu cores and 8 cpu threads. But it is configured with 2x 1TB SATA software raid 1.

But when i run gzip vs pigz tests at even level 1 compression on 5.5 GB sized mysqldump sql backup file, i get unusual results.

1. gzip averages around 46 seconds
2. pigz averages around 44 seconds and only hits max 90-125% cpu usage even if specifically use -p2 to -p4 options ?

example

gzip -3 backup.sql
real: 46.91s cpu: 78% maxmem: 3424 KB cswaits: 33532

pigz -3 backup.sql
real: 44.27s cpu: 93% maxmem: 32816 KB cswaits: 481219

Are there any known issues with software raid or CentOS 6.3 or pigz 2.2.4/2.2.5 which would cause this ?

top - 08:45:22 up 5 days,  9:43,  2 users,  load average: 0.00, 0.03, 0.08
Tasks: 314 total,   1 running, 313 sleeping,   0 stopped,   0 zombie
Cpu0  :  0.4%us,  0.2%sy,  0.1%ni, 97.8%id,  1.5%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  :  0.2%us,  0.2%sy,  0.2%ni, 99.2%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.1%us,  0.1%sy,  0.2%ni, 99.4%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.1%us,  0.1%sy,  0.1%ni, 99.7%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  0.2%us,  0.3%sy,  0.3%ni, 98.8%id,  0.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  0.1%us,  0.1%sy,  0.3%ni, 99.4%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  0.1%us,  0.1%sy,  0.3%ni, 99.4%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  0.1%us,  0.1%sy,  0.4%ni, 99.3%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16324444k total,  6658576k used,  9665868k free,    25480k buffers
Swap: 18563064k total,        0k used, 18563064k free,   927772k cached

CentOS 6.3 64bit
2.6.32-279.9.1.el6.x86_64

cat /proc/mdstat
Personalities : [raid1]
md127 : active raid1 sda[1] sdb[0]
976759808 blocks super external:/md0/0 [2/2] [UU]

md0 : inactive sdb1 sda0
5288 blocks super external:imsm

unused devices:

I tried with lbzip2 to compare and that correctly uses nearly all 8 cpu threads

lbzip2 -1 backup.sql
real: 264.75s cpu: 787% maxmem: 112752 KB cswaits: 118854

with plzip around 560% cpu usage

plzip -1 backup.sql
real: 47.54s cpu: 560% maxmem: 667488 KB cswaits: 49565

then compare gzip level 1

gzip -1 backup.sql
real: 53.46s cpu: 66% maxmem: 3424 KB cswaits: 34327

top stats while gzip running, i notice both gzip and pigz momentarily goes into D state ?

top - 09:00:17 up 5 days,  9:58,  2 users,  load average: 0.82, 2.07, 1.68
Tasks: 319 total,   2 running, 316 sleeping,   0 stopped,   1 zombie
Cpu0  : 39.9%us,  3.7%sy,  0.0%ni,  0.3%id, 55.4%wa,  0.0%hi,  0.7%si,  0.0%st
Cpu1  :  1.0%us,  0.7%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  :  0.3%us,  0.3%sy,  0.0%ni, 99.0%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  0.7%us,  0.7%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  7.9%us,  1.0%sy,  0.0%ni, 55.6%id, 35.4%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  1.0%us,  0.7%sy,  0.0%ni, 98.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  :  2.3%us,  0.7%sy,  0.0%ni, 97.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  :  4.3%us,  1.3%sy,  0.0%ni, 94.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16324444k total,  8481500k used,  7842944k free,    28816k buffers
Swap: 18563064k total,        0k used, 18563064k free,  2717080k cached

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                         
19582 root      20   0  4292  800  408 D 40.9  0.0   0:08.19 gzip 

yum list installed zlib -q
Installed Packages
zlib.x86_64                                               1.2.3-27.el6

We'd be interested in using pigz as a library in our application and would want to support Windows, Linux, and Mac with a uniform build system that supports native compilers (e.g. Visual Studio, not mingw). CMake has been our go-to solution for that so it would be great if pigz had CMake support.

Has anyone looked into CMake for pigz before?

[root@testing pigz-2.3.1]# make CPPFLAGS="-l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/"
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o pigz.o pigz.c
pigz.c: In function ‘compress_thread’:
pigz.c:1426: warning: ‘temp’ may be used uninitialized in this function
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o yarn.o yarn.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/deflate.o zopfli/deflate.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/blocksplitter.o zopfli/blocksplitter.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/tree.o zopfli/tree.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/lz77.o zopfli/lz77.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/cache.o zopfli/cache.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/hash.o zopfli/hash.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/util.o zopfli/util.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/squeeze.o zopfli/squeeze.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-128/lib/ -I/usr/local/zlib-128/include/ -c -o zopfli/katajainen.o zopfli/katajainen.c
cc -o pigz pigz.o yarn.o zopfli/deflate.o zopfli/blocksplitter.o zopfli/tree.o zopfli/lz77.o zopfli/cache.o zopfli/hash.o zopfli/util.o zopfli/squeeze.o zopfli/katajainen.o -lpthread -lz -lm
pigz.o: In function compress_thread': pigz.c:(.text+0x3f28): undefined reference todeflatePending'
pigz.c:(.text+0x42b7): undefined reference to deflatePending' pigz.o: In functionsingle_compress':
pigz.c:(.text+0x6b60): undefined reference to deflatePending' pigz.c:(.text+0x6b95): undefined reference todeflatePending'
collect2: ld returned 1 exit status
make: *** [pigz] Error 1

It has no problems with zlib 1.2.3:
[root@testing pigz-2.3.1]# make CPPFLAGS="-l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/"
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o pigz.o pigz.c
pigz.c: In function ‘compress_thread’:
pigz.c:1426: warning: ‘temp’ may be used uninitialized in this function
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o yarn.o yarn.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/deflate.o zopfli/deflate.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/blocksplitter.o zopfli/blocksplitter.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/tree.o zopfli/tree.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/lz77.o zopfli/lz77.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/cache.o zopfli/cache.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/hash.o zopfli/hash.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/util.o zopfli/util.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/squeeze.o zopfli/squeeze.c
cc -O3 -Wall -Wextra -l/usr/local/zlib-123/lib/ -I/usr/local/zlib-123/include/ -c -o zopfli/katajainen.o zopfli/katajainen.c
cc -o pigz pigz.o yarn.o zopfli/deflate.o zopfli/blocksplitter.o zopfli/tree.o zopfli/lz77.o zopfli/cache.o zopfli/hash.o zopfli/util.o zopfli/squeeze.o zopfli/katajainen.o -lpthread -lz -lm
ln -f pigz unpigz

The github-generated tarballs can and do change over time (as they're generated on demand using git-archive) so they're not useful for distributions which perform integrity checks on downloaded tarballs.

Only the latest tarball is available on zlib.net, for example https://zlib.net/pigz/pigz-2.4.tar.gz is present but https://zlib.net/pigz/pigz-2.3.4.tar.gz doesn't exist anymore.

Please consider uploading the generated tarballs to github and associating them with the releases as a persistent archive of the official tarballs.

Hello!

If I download your code and compile and install it, and make some judicious use of symlinks, can I make my system run pigz whenever it tries to run gzip? And if so would that work well? Also: any chance of your tool or another multi-threaded zipper replacing gzip, out of the box so to speak, on any Linux distributions? Thanks.

Trying to decompress a backup (tar.gz) with pigz and getting this error. When I disable parallelization with -dp1, I still get the same error. However, I can decompress the file successfully with gunzip or tar -xvzf. pigz is the only tool that fails on decompression of this file. However, all the other backups decompress with pigz, it is just this particular backup file. The fact that other decompression algorithms work fine makes me wonder if this might be a bug in pigz? The pigz version is 2.1.6.

On pigz -drv:

c/ch/chillsonicfanon_pages_full.xml.gz to c/ch/chillsonicfanon_pages_full.xml pigz abort: corrupted input -- invalid deflate data: c/ch/chillsonicfanon_pages_full.xml.gz

Correct, because gunzip agrees:

$ gunzip c/ch/chillsonicfanon_pages_full.xml.gz
gzip: c/ch/chillsonicfanon_pages_full.xml.gz: unexpected end of file

But it's quite annoying because I expect the directory to be all uncompressed and instead I have all the following files still compressed.

There doesn't seem to be a tag for 2.3.1 release https://github.com/madler/pigz/tags or is this version only available in master ?

When running pigz without a terminal (for example via crontab) and a compressed file already exists, it's not possible for pigz to ask the user to overwrite or not. This causes pigz to fail with an error message:

pigz: abort: write error on 001.txt.gz (Inappropriate ioctl for device)

It would be more useful with something along the lines of

pigz: abort: compressed file exists: 001.txt.gz
or perhaps better?
pigz: skipping 001.txt: compressed file exists: 001.txt.gz

It would also be useful to have a --no-overwrite option.

pigz 2.4, Fedora 27, x64

Hi

I understand that compression algorithms are CPU and IO bounded however CPU is being utilised much more. Is there any performance gains with pigz since reading data will make the IO device the actual bottleneck? It sounds more logical to gain some performance on SSDs but not on normal HDs? Could you please elaborate?

on AIX to compress files larger than 2G the opne/stat calls need to be replaced with stat64/open64. I will try to attach a pull request with a fix.

Is there a way to compess a directory without using tar' but with a piece of code?
I suggest to compress one file and then a possibility to copy other files into the archive ? Am i wrong?

If not, how to procceed please?

please: mgerdts@fe9c1ed

I have been using pigz as a system-wide replacement to gzip for a while (as a gzip symlink earlier in PATH). I have been hitting an issue that pigz compressed files are not checksum-equal to those compressed by gzip.
In summary, for a project I work with, sha256 hashes are calculated for compressed tarballs, and I ended up submitting some hashes for tarballs I compressed with pigz, which resulted in them being different for users using gzip.
(for reference: https://patchwork.ozlabs.org/patch/975483/ )

I assume there are several reasons for why they are not exactly the same, but is there a way to make it so? I tried a few values for blocksize with pigz, used --no-name to avoid timestamps, but that didn't help. The number of threads doesn't seem to make a difference. Is there some parameter set or condition to make the result exactly the same as it would be with gzip?

A zlib like library for pigz is wanted

I tried to compile pigz on windows (mingw) and successfully done it with some effort but the solution turned out to be simple - implement fsync() using windows native functions. I attach patch file so you can check it out. Hope you will find it useful. There are two versions of fsync() - one using FlushFileBuffers() and second using _commit(). You can choose whichever you find appropriate- there seem to be no difference in performance. Files are somewhat excessively commented so should be no problem reading them.
Both versions of fsync() work well - I tested them on over 6k+ files of all sizes - from 1B to 6GiB. Compressed, tested with gzip and pigz and decompressed to make sure they are not corrupted and they were fine.
Hope you will like it and use it in future release.
pigz-2.4.win_fsync-2.diff.gz

I'm having an issue when trying to create a zip archive. I have a specific file that when I compress using the -K option cannot be uncompressed afterwards. I get a CRC failure message.
I don't receive any error messages during compression.
I can compress the file fine using 7zip.
I can compress to gz format using pigz without issue.
The file is over 5 gigabytes.
I'm using RHEL v 6.8 (Santiago) on a x86_64 machine.

I assume that pigz 2.3.4 is intended to be a drop-in replacement for gzip. However, it does not seem to be compatible with the latest version of GNU tar (1.29), at least when tar is configured at compile time to use pigz as its gzip program. In particular, using pigz causes the gzip test (Test 26) of the official test suite to fail. Steps to reproduce:

wget -O - https://ftp.gnu.org/gnu/tar/tar-1.29.tar.bz2 | tar xjv
cd tar-1.29
./configure --with-gzip=pigz
make
cd tests
make check-full

Results:

## ------------------------ ##
## GNU tar 1.29 test suite. ##
## ------------------------ ##
26. gzip.at:24: testing gzip ...
./gzip.at:29:

cat /dev/null | gzip - > /dev/null 2>&1 || exit 77

tar xfvz /dev/null 2>err
RC=$?
sed -n '/^tar:/p' err >&2
exit $RC

--- -   2017-07-24 13:28:13.545914213 +0200
+++ /tmp/tar-1.29/tests/testsuite.dir/at-groups/26/stderr       2017-07-24 13:28:13.537509112 +0200
@@ -1,3 +1 @@
-tar: Child returned status 1
-tar: Error is not recoverable: exiting now
 
./gzip.at:29: exit code was 0, expected 2
26. gzip.at:24:  FAILED (gzip.at:29)

The test is expecting the compression child process to fail, but pigz completes successfully. Compare:

tar xfv /dev/null --use-compress-program=gzip; echo Return value: $?

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
Return value: 2

tar xfv /dev/null --use-compress-program=pigz; echo Return value: $?
Return value: 0

Commit 28d11df breaks the build on Cygwin and Windows because PE platforms mandate the equivalent of -Wl,--as-needed. Please revert.

that can be a deadlock. via tsan (going by memory, I hope I am not mis-remembering)

pigz.c:1458

    possess(space->use);
    use = peek_lock(space->use);
    assert(use != 0);
    if (use == 1) {
        pool = space->pool;
        possess(pool->have);

pigz.c:1399

    possess(pool->have);
    if (pool->limit == 0)
        wait_for(pool->have, NOT_TO_BE, 0);

    // if a space is available, pull it from the list and return it
    if (pool->head != NULL) {
        space = pool->head;
        possess(space->use);

Hi
Is there a way to change the header of ziped file in order to be able to decompress it only with pigz decompresser and not with winrar, 7zip or any other archiver ?

Regards

I just started to use pigz and realized that "pigz -d" of "sample.tgz" returns "sample" instead of what gzip returns, which is "sample.tar". Can you change this to behave like gzip? Using pigz 2.2.4 and gzip 1.3.12.

Thanks!

$ uname -a
Linux zombie 3.18.1+ #5 SMP PREEMPT Sun Dec 21 13:32:40 CET 2014 x86_64 GNU/Linux
$ make
cc -O3 -Wall -Wextra -c -o pigz.o pigz.c
pigz.c: In function ‘cut_short’:
pigz.c:824:9: warning: suggest braces around empty body in an ‘if’ statement [-Wempty-body]
Trace(("termination by user"));
^
pigz.c: In function ‘infchk’:
pigz.c:3066:24: error: ‘EFTYPE’ undeclared (first use in this function)
throw(EFTYPE, "%s: corrupted -- invalid deflate data (%s)",
^
pigz.c:3066:24: note: each undeclared identifier is reported only once for each function it appears in
pigz.c: In function ‘unlzw’:
pigz.c:3204:20: error: ‘EFTYPE’ undeclared (first use in this function)
throw(EFTYPE, "%s: lzw premature end", g.inf);
^
pigz.c: In function ‘process’:
pigz.c:3458:28: error: ‘EFTYPE’ undeclared (first use in this function)
throw(EFTYPE, "%s too large -- "
^
: recipe for target 'pigz.o' failed
make: *** [pigz.o] Error 1

Hi

Many days ago i tried to look for a way to modify the code in order to compress a diretory !

Have you an idea on how to process please ?

Thank's.

Solaris 10 with GCC
bash-3.2# gcc -v
Reading specs from /usr/sfw/lib/gcc/sparc-sun-solaris2.10/3.4.3/specs
Configured with: /sfw10/builds/build/sfw10-patch/usr/src/cmd/gcc/gcc-3.4.3/configure --prefix=/usr/sfw --with-as=/usr/ccs/bin/as --without-gnu-as --with-ld=/usr/ccs/bin/ld --without-gnu-ld --enable-languages=c,c++ --enable-shared
Thread model: posix
gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath)

I fixed my build issue while writing this up.. Persons compiling with gcc under solaris 10 may find that running make does not find the deflate.o, blocksplitter.o, tree.o, and other files. The solution was to copy them from the main build directory to zopfli/

bash-3.2# make
gcc -O3 -Wall -Wextra -c zopfli/deflate.c
gcc -O3 -Wall -Wextra -c zopfli/blocksplitter.c
gcc -O3 -Wall -Wextra -c zopfli/tree.c
gcc -O3 -Wall -Wextra -c zopfli/lz77.c
gcc -O3 -Wall -Wextra -c zopfli/cache.c
gcc -O3 -Wall -Wextra -c zopfli/hash.c
gcc -O3 -Wall -Wextra -c zopfli/util.c
gcc -O3 -Wall -Wextra -c zopfli/squeeze.c
gcc -O3 -Wall -Wextra -c zopfli/katajainen.c
gcc -o pigz pigz.o yarn.o zopfli/deflate.o zopfli/blocksplitter.o zopfli/tree.o zopfli/lz77.o zopfli/cache.o zopfli/hash.o zopfli/util.o zopfli/squeeze.o zopfli/katajainen.o -lpthread -lz
gcc: zopfli/deflate.o: No such file or directory
gcc: zopfli/blocksplitter.o: No such file or directory
gcc: zopfli/tree.o: No such file or directory
gcc: zopfli/lz77.o: No such file or directory
gcc: zopfli/cache.o: No such file or directory
gcc: zopfli/hash.o: No such file or directory
gcc: zopfli/util.o: No such file or directory
gcc: zopfli/squeeze.o: No such file or directory
gcc: zopfli/katajainen.o: No such file or directory
*** Error code 1
make: Fatal error: Command failed for target `pigz'

If I move the .o files from the root of the build to zopfli/
and re-attempt the build I get:

bash-3.2# cp .o zopfli/
bash-3.2# make
gcc -o pigz pigz.o yarn.o zopfli/deflate.o zopfli/blocksplitter.o zopfli/tree.o zopfli/lz77.o zopfli/cache.o zopfli/hash.o zopfli/util.o zopfli/squeeze.o zopfli/katajainen.o -lpthread -lz
Undefined first referenced
symbol in file
log zopfli/tree.o
ld: fatal: symbol referencing errors. No output written to pigz
collect2: ld returned 1 exit status
** Error code 1
make: Fatal error: Command failed for target `pigz'

From there it seems to just be the -LM error to fix.
changed:
$(CC) -o pigz $^ -lpthread -lz
to:
$(CC) -o pigz $^ -lpthread -lz -lm

Are there any tests that go along with pigz?

Hi,

In a previous issue, you swiftly updated the code when this issue was reported, however the code appears to have regressed ?

cc -O3 -Wall -Wextra -c -o pigz.o pigz.c
pigz.c: In function ‘copymeta’:
pigz.c:3051:5: warning: ignoring return value of ‘chown’, declared with attribute warn_unused_result [-Wunused-result]
pigz.c: In function ‘compress_thread’:
pigz.c:1118:21: warning: ‘temp’ may be used uninitialized in this function [-Wuninitialized]
pigz.c:1283:19: note: ‘temp’ was declared here
cc -O3 -Wall -Wextra -c -o yarn.o yarn.c
cc -O3 -Wall -Wextra -c -o zopfli/deflate.o zopfli/deflate.c
cc -O3 -Wall -Wextra -c -o zopfli/blocksplitter.o zopfli/blocksplitter.c
cc -O3 -Wall -Wextra -c -o zopfli/tree.o zopfli/tree.c
cc -O3 -Wall -Wextra -c -o zopfli/lz77.o zopfli/lz77.c
cc -O3 -Wall -Wextra -c -o zopfli/cache.o zopfli/cache.c
cc -O3 -Wall -Wextra -c -o zopfli/hash.o zopfli/hash.c
cc -O3 -Wall -Wextra -c -o zopfli/util.o zopfli/util.c
cc -O3 -Wall -Wextra -c -o zopfli/squeeze.o zopfli/squeeze.c
cc -O3 -Wall -Wextra -c -o zopfli/katajainen.o zopfli/katajainen.c
cc -o pigz pigz.o yarn.o zopfli/deflate.o zopfli/blocksplitter.o zopfli/tree.o zopfli/lz77.o zopfli/cache.o zopfli/hash.o zopfli/util.o zopfli/squeeze.o zopfli/katajainen.o -lpthread -lz
zopfli/tree.o: In function CalculateEntropy': tree.c:(.text+0x193): undefined reference tolog'
tree.c:(.text+0x23e): undefined reference to log' tree.c:(.text+0x29b): undefined reference tolog'
collect2: ld returned 1 exit status
make: *** [pigz] Error 1

This would be cool to get us closer to being a drop-in replacement for gnu gzip

Is there a way to use pigz with a Zip File with multiples text files inside ?

Example of my testing
pigz -dc file.zip | awk '{print $3}' | sort | uniq -c
pigz: warning: file.zip: entries after the first were ignored

Only one file seem to be extract.

pigz returns 0 as exit code even if the file extraction is failed. Is it a bug or am I missing something here?

deepak@my-ubuntu:~$ pigz -d -c file_not_zipped
pigz: skipping: file_not_zipped is not compressed
deepak@my-ubuntu:~$ echo $?
0

I noticed that when pigz is given a file name that doesn't exist, it skips the file, printing

pigz: skipping: foo does not exist

and exits with status 0. I can imagine some uses for this behavior, but it seems more useful to me to treat this an error and exit non-0.

Was the current behavior chosen to handle some common usage pattern? If not, what are your thoughts on changing pigz to exit non-0 in this case? It looks like gzip exits with status 1 after printing an error message.

I was trying to tar.gz 3 million files.
Midway I cancelled the operation because of reasons.
300k files went awol.

Seriously though, I feel that having if a program changes the system state by default is not something that is expected by default.

Hello,

I was looking for a great program that uses Zopfli, but pigz did not mention it anywhere in the readme or manpage. I had to look at the sources to realize that -11 uses the Zopfli alhorithm for compression.

Could you add this tidbit of information to the manpage and possibly --help output aswell? This would be much appreciated :)

Thanks!

When pigz is compiled with ThreadSanitizer, the detector reports a lock-order-inversion (potential deadlock) warning by executing:
TSAN_OPTIONS=abort_on_error=1:halt_on_error=0:symbolize=1:report_thread_leaks=1:report_destroy_locked=1:report_signal_unsafe=1:second_deadlock_stack=1 ./pigz-tsan/pigz --best -p 4 -c -b 32 ./pigz-reports/test.in >/dev/null 2> pigz-reports/test.tsan.txt
pigz-reports.zip

It seems that there might be a cycle lock when invoking possess function at yarn.c:115 multiple times.

==================                                                                                                                                                                                                                             
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6829)                                                                                                                                                                 
  Cycle in lock order graph: M9 (0x7b1800000240) => M4 (0x7b18000000c0) => M9                                                                                                                                                                  
                                                                                                                                                                                                                                               
  Mutex M4 acquired here while holding mutex M9 in thread T1:                                                                                                                                                                                  
    #0 pthread_mutex_lock <null> (pigz+0x42f3e5)                                                                                                                                                                                               
    #1 possess /home/hongxu/work/pigz/pigz-tsan/yarn.c:115:16 (pigz+0x4d7fc8)                                                                                                                                                                  
    #2 drop_space /home/hongxu/work/pigz/pigz-tsan/pigz.c:1462:9 (pigz+0x4d5526)                                                                                                                                                               
    #3 write_thread /home/hongxu/work/pigz/pigz-tsan/pigz.c:1913:13 (pigz+0x4d11f2)                                                                                                                                                            
    #4 ignition /home/hongxu/work/pigz/pigz-tsan/yarn.c:253:5 (pigz+0x4d8738)                                                                                                                                                                  
                                                                                                                                                                                                                                               
  Mutex M9 previously acquired by the same thread here:                                                                                                                                                                                        
    #0 pthread_mutex_lock <null> (pigz+0x42f3e5)                                                                                                                                                                                               
    #1 possess /home/hongxu/work/pigz/pigz-tsan/yarn.c:115:16 (pigz+0x4d7fc8)                                                                                                                                                                  
    #2 drop_space /home/hongxu/work/pigz/pigz-tsan/pigz.c:1457:5 (pigz+0x4d5481)                                                                                                                                                               
    #3 write_thread /home/hongxu/work/pigz/pigz-tsan/pigz.c:1913:13 (pigz+0x4d11f2)                                                                                                                                                            
    #4 ignition /home/hongxu/work/pigz/pigz-tsan/yarn.c:253:5 (pigz+0x4d8738)                                                                                                                                                                  
                                                                                                                                                                                                                                               
  Mutex M9 acquired here while holding mutex M4 in main thread:                                                                                                                                                                                
    #0 pthread_mutex_lock <null> (pigz+0x42f3e5)                                                                                                                                                                                               
    #1 possess /home/hongxu/work/pigz/pigz-tsan/yarn.c:115:16 (pigz+0x4d7fc8)                                                                                                                                                                  
    #2 get_space /home/hongxu/work/pigz/pigz-tsan/pigz.c:1406:9 (pigz+0x4d1861)                                                                                                                                                                
    #3 parallel_compress /home/hongxu/work/pigz/pigz-tsan/pigz.c:2033:20 (pigz+0x4c91c6)                                                                                                                                                       
    #4 process /home/hongxu/work/pigz/pigz-tsan/pigz.c:3932:9 (pigz+0x4bccf5)                                                                                                                                                                  
    #5 main /home/hongxu/work/pigz/pigz-tsan/pigz.c:4420:17 (pigz+0x4b889b)                                                                                                                                                                    
                                                                                                                                                                                                                                               
  Mutex M4 previously acquired by the same thread here:                                                                                                                                                                                        
    #0 pthread_cond_wait <null> (pigz+0x4456d1)                                                                                                                                                                                                
    #1 wait_for /home/hongxu/work/pigz/pigz-tsan/yarn.c:154:24 (pigz+0x4d825a)                                                                                                                                                                 
    #2 get_space /home/hongxu/work/pigz/pigz-tsan/pigz.c:1401:9 (pigz+0x4d17fc)                                                                                                                                                                
    #3 parallel_compress /home/hongxu/work/pigz/pigz-tsan/pigz.c:2033:20 (pigz+0x4c91c6)                                                                                                                                                       
    #4 process /home/hongxu/work/pigz/pigz-tsan/pigz.c:3932:9 (pigz+0x4bccf5)                                                                                                                                                                  
    #5 main /home/hongxu/work/pigz/pigz-tsan/pigz.c:4420:17 (pigz+0x4b889b)                                                                                                                                                                    
                                                                                                                                                                                                                                               
  Thread T1 (tid=6831, running) created by main thread at:                                                                                                                                                                                     
    #0 pthread_create <null> (pigz+0x428b36)                                                                                                                                                                                                   
    #1 launch /home/hongxu/work/pigz/pigz-tsan/yarn.c:288:16 (pigz+0x4d8561)                                                                                                                                                                   
    #2 parallel_compress /home/hongxu/work/pigz/pigz-tsan/pigz.c:2009:15 (pigz+0x4c8fab)                                                                                                                                                       
    #3 process /home/hongxu/work/pigz/pigz-tsan/pigz.c:3932:9 (pigz+0x4bccf5)                                                                                                                                                                  
    #4 main /home/hongxu/work/pigz/pigz-tsan/pigz.c:4420:17 (pigz+0x4b889b)                                                                                                                                                                    
                                                                                                                                                                                                                                               
SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/home/hongxu/work/pigz/pigz-tsan/pigz+0x42f3e5) in pthread_mutex_lock                                                                                                     
==================                                                                                                                                                                                                                             
==================                                                                                                                                                                                                                             
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=6829)                                                                                                                                                                 
  Cycle in lock order graph: M14 (0x7b1800003000) => M5 (0x7b1800000120) => M14                                                                                                                                                                
                                                                                                                                                                                                                                               
  Mutex M5 acquired here while holding mutex M14 in thread T1:                                                                                                                                                                                 
    #0 pthread_mutex_lock <null> (pigz+0x42f3e5)                                                                                                                                                                                               
    #1 possess /home/hongxu/work/pigz/pigz-tsan/yarn.c:115:16 (pigz+0x4d7fc8)                                                                                                                                                                  
    #2 drop_space /home/hongxu/work/pigz/pigz-tsan/pigz.c:1462:9 (pigz+0x4d5526)                                                                                                                                                               
    #3 write_thread /home/hongxu/work/pigz/pigz-tsan/pigz.c:1920:13 (pigz+0x4d133c)                                                                                                                                                            
    #4 ignition /home/hongxu/work/pigz/pigz-tsan/yarn.c:253:5 (pigz+0x4d8738)                                                                                                                                                                  
                                                                                                                                                                                                                                               
  Mutex M14 previously acquired by the same thread here:                                                                                                                                                                                       
    #0 pthread_mutex_lock <null> (pigz+0x42f3e5)                                                                                                                                                                                               
    #1 possess /home/hongxu/work/pigz/pigz-tsan/yarn.c:115:16 (pigz+0x4d7fc8)                                                                                                                                                                  
    #2 drop_space /home/hongxu/work/pigz/pigz-tsan/pigz.c:1457:5 (pigz+0x4d5481)                                                                                                                                                               
    #3 write_thread /home/hongxu/work/pigz/pigz-tsan/pigz.c:1920:13 (pigz+0x4d133c)                                                                                                                                                            
    #4 ignition /home/hongxu/work/pigz/pigz-tsan/yarn.c:253:5 (pigz+0x4d8738)                                                                                                                                                                  
                                                                                                                                                                                                                                               
  Mutex M14 acquired here while holding mutex M5 in thread T2: 
...
...
...                   

Hi!

I tried to compile the current version of pigz (2.3.3) on Ubuntu 14.04 and bumped into the following error:

cc -lz -o pigz pigz.o yarn.o try.o zopfli/src/zopfli/deflate.o zopfli/src/zopfli/blocksplitter.o zopfli/src/zopfli/tree.o zopfli/src/zopfli/lz77.o zopfli/src/zopfli/cache.o zopfli/src/zopfli/hash.o zopfli/src/zopfli/util.o zopfli/src/zopfli/squeeze.o zopfli/src/zopfli/katajainen.o -lpthread -lm pigz.o: In function 'outb_check': pigz.c:(.text+0x4b9): undefined reference to 'crc32' pigz.c:(.text+0x531): undefined reference to 'adler32' [...]

I can see that zlib linkage flag -lz is put at the very beginning of the cc command, because $(LDFLAGS) is put there in the Makefile:

pigz: pigz.o yarn.o try.o ${ZOPFLI}deflate.o ${ZOPFLI}blocksplitter.o ${ZOPFLI}tree.o ${ZOPFLI}lz77.o ${ZOPFLI}cache.o ${ZOPFLI}hash.o ${ZOPFLI}util.o ${ZOPFLI}squeeze.o ${ZOPFLI}katajainen.o $(CC) $(LDFLAGS) -o pigz $^ -lpthread -lm ln -f pigz unpigz

But, as I understand, -lz should be put at the very end of the command on Linux. Makefile with the following command works without a problem:

pigz: pigz.o yarn.o try.o ${ZOPFLI}deflate.o ${ZOPFLI}blocksplitter.o ${ZOPFLI}tree.o ${ZOPFLI}lz77.o ${ZOPFLI}cache.o ${ZOPFLI}hash.o ${ZOPFLI}util.o ${ZOPFLI}squeeze.o ${ZOPFLI}katajainen.o $(CC) -o pigz $^ -lpthread -lm $(LDFLAGS) ln -f pigz unpigz

Probably, it would be great to add some check to the Makefile to put $(LDFLAGS) to the end of the cc command when compiling on Linux.

Best regards,
Svyatoslav

Hi,

please check if possible.

gcc -g -O2 -fdebug-prefix-map=/pigz-2.4=. -fstack-protector-strong -Wformat -Werror=format-security -c zopfli/src/zopfli/deflate.c

zopfli/src/zopfli/deflate.c: In function ‘OptimizeHuffmanForRle’:
zopfli/src/zopfli/deflate.c:399:16: warning: argument 1 value ‘18446744073709551612’ exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
good_for_rle = (int*)malloc(length * sizeof(int));

In file included from zopfli/src/zopfli/zopfli.h:24:0,
              from zopfli/src/zopfli/deflate.h:28,
              from zopfli/src/zopfli/deflate.c:20:
/usr/include/stdlib.h:443:14: note: in a call to allocation function ‘malloc’ declared here
extern void *malloc (size_t __size) __THROW __attribute_malloc__ __wur;
           ^~~~~~

$ gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian 7.2.0-18' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.2.0 (Debian 7.2.0-18) 

There is a project zlib-ng to do a faster single-threaded zlib.

There's interest in seeing how the pigz code might be sped up by incorporating code or libraries from zlib-ng, expressed at https://github.com/Dead2/zlib-ng/issues/96

Cross-linking those two ideas, here; I have not started to try to see how the build would change to use that other library.

I was just trying out whether an in-place compression is possible using pigz -S '' - it turns out pigz just follows its usual workflow, i.e. it compresses and then unlinks the original. Which in this case is the same as the target.

$ echo "hello world" > testfile
$ pigz -S '' -f testfile && echo OK
OK
$ ls
$

No error return code and even worse, the data is gone.

For the record, I then tried gzip and it ignores -S '' and behaves as if -S wasn't specified.

If I decompress a file with pigz -d foo.gz it only uses one core, on my multicore system. Compressing uses all my cores, but decompressing does not. Is this by design? I'm trying to compare decompression speeds or pigz vs vs pbzip2 vs lz4.

pbzip2 does mutli-threaded decompression so I assumed pigz would too.

:cat /tmp/d8.qcow.lz4 | pv | lz4 -d > /dev/null
936MiB 0:00:06 [ 152MiB/s]

:cat /tmp/d8.qcow.gz | pv | pigz -d > /dev/null
705MiB 0:00:12 [  56MiB/s]

:cat /tmp/d8.qcow.bz2 | pv | pbzip2 -d > /dev/null
637MiB 0:00:04 [ 129MiB/s]

There are several open issues and last commit is from 2 Oct 2016.

In original gzip

gzip -c -d foo.tar.gz > bar.tar

and

gzip -d foo.tar.gz -c > bar.tar

work identical: keep foo.tar.gz unchanged, create bar.tar as uncompressed .tar

pigz -c -d foo.tar.gz > bar.tar

work as needed

pigz -d foo.tar.gz -c > bar.tar

ignore "-c", bar.tar empty foo.tar.gz became foo.tar

see ib/xarchiver#40

Attached is a zip file that other utilities can decompress, but pigz complains:

$ pigz -d test.txt.zip
pigz: test.txt.zip is not compressed -- skipping
$ zip -T test.txt.zip
test of test.txt.zip OK

Is something wrong with this file or my usage of pigz?

test.txt.zip

I'm creating a gunzipped tarball of ~3M files with 720 GB of an ext4 partition with

sudo tar cf - to_compress/ | pigz -p 16 > /path/to/compressed.tar.gz

After > 500 GB pigz gets stuck using > 90 % of CPU with no observable I/O activity (iotop). Shouldn't gunzip and therefore also pigz read data continuously? Changing the number threads doesn't change (I only changed once because the test takes some time...). Are you testing for such large input data sets?

experienced with pigz 1c9c03b and 2.2.4 on Ubuntu 13.10.

pigz's -n option causes the file name not to be stored in the header, while gzip's -n option causes both the file name and the mod time not to be stored. I see that pigz has a -T option to not store mod time, but gzip doesn't support this flag.

It would be nice if you could install pigz as a drop-in gzip replacement, but currently scripts which call gzip that want to create .gz files reproducibly with no timestamp just pass -n, which doesn't omit timestamps with pigz.

My suggestion is to change pigz's -n option to match gzip (don't store file name and mod time), and maybe introduce a new option (like -T) which omits only file name.

I think it expects sensible handling of the return code...

cc -O3 -Wall -Wextra -D_FORTIFY_SOURCE=2  -c -o zopfli/src/zopfli/katajainen.o zopfli/src/zopfli/katajainen.c
pigz.c: In function ‘copymeta’:
pigz.c:3399:5: warning: ignoring return value of ‘chown’, declared with attribute warn_unused_result [-Wunused-result]
     (void)chown(to, st.st_uid, st.st_gid);
     ^

I'm using pigz to speed up transfer of large files from embedded device to server. It goes like this:

pigz -1 -p2 -c | curl -T - $url

My goal is to minimize total time of file compression&transmission. Problem is that I don't know beforehand what bandwidth is available for upload, and it can be quite small (e.g., 1 Mbit/sec). In this case it would be beneficial to use higher compression level.

Would it make sense if pigz supported adaptive compression level based on average write speed (in this case, to stdout)? E.g., start with 1 and increase it if write throughput is lower than compression throughput.

Hi tried source compile of pigz v2.3 but seems getting an error at make stage as below

make
cc -O3 -Wall -Wextra   -c -o pigz.o pigz.c
pigz.c: In function ‘compress_thread’:
pigz.c:1283: warning: ‘temp’ may be used uninitialized in this function
cc -O3 -Wall -Wextra   -c -o yarn.o yarn.c
cc -O3 -Wall -Wextra   -c -o zopfli/deflate.o zopfli/deflate.c
cc -O3 -Wall -Wextra   -c -o zopfli/blocksplitter.o zopfli/blocksplitter.c
cc -O3 -Wall -Wextra   -c -o zopfli/tree.o zopfli/tree.c
cc -O3 -Wall -Wextra   -c -o zopfli/lz77.o zopfli/lz77.c
cc -O3 -Wall -Wextra   -c -o zopfli/cache.o zopfli/cache.c
cc -O3 -Wall -Wextra   -c -o zopfli/hash.o zopfli/hash.c
cc -O3 -Wall -Wextra   -c -o zopfli/util.o zopfli/util.c
cc -O3 -Wall -Wextra   -c -o zopfli/squeeze.o zopfli/squeeze.c
cc -O3 -Wall -Wextra   -c -o zopfli/katajainen.o zopfli/katajainen.c
cc -o pigz pigz.o yarn.o zopfli/deflate.o zopfli/blocksplitter.o zopfli/tree.o zopfli/lz77.o zopfli/cache.o zopfli/hash.o zopfli/util.o zopfli/squeeze.o zopfli/katajainen.o -lpthread -lz
zopfli/tree.o: In function `CalculateEntropy':
tree.c:(.text+0x75): undefined reference to `log'
tree.c:(.text+0x11a): undefined reference to `log'
tree.c:(.text+0x16b): undefined reference to `log'
collect2: ld returned 1 exit status
make: *** [pigz] Error 1

On embedded systems zopfli is prohibitively slow, so it just bloats pigz binary. It would be nice to have an option to exclude it from build.