boludoz / lz4 Goto Github PK
View Code? Open in Web Editor NEWAutomatically exported from code.google.com/p/lz4
License: Other
Automatically exported from code.google.com/p/lz4
License: Other
unpack source distro:
python setup.py build
Traceback (most recent call last):
File "setup.py", line 13, in <module>
long_description=open('README.rst', 'r').read(),
IOError: [Errno 2] No such file or directory: 'README.rst'
Original issue reported on code.google.com by [email protected]
on 14 Mar 2012 at 5:36
In some cases, LZ4_uncompress does not return error codes when the input is
invalid.
For example uncompressing [16, 42, 1, 0, 80, 42, 42, 42, 42, 42, 42] (which is
a compressed form of an array of length 10 whose elements are all 42) with
osize == 11 won't return an error, although the uncompressed length of the
array is 10.
Additionally, LZ4 does not return errors when ref == op (dec == 0).
Original issue reported on code.google.com by [email protected]
on 29 Oct 2012 at 9:21
Attachments:
There are two API calls that work with context - LZ4_compressCtx and
LZ4_compress64kCtx. But there is not API to free context at the end.
Original issue reported on code.google.com by specialforest
on 29 Jun 2012 at 2:44
Any ideas for improving compress performance on ARM? I'm running some
benchmarks on an OpenVPN packet dataset (IPv4 packets, < 1500 bytes each,
reflecting real-world web browsing sessions) and LZ4 is taking twice as much
time as LZO to achieve roughly comparable compression ratios.
James
Original issue reported on code.google.com by [email protected]
on 30 Jan 2012 at 9:24
Hi,
I am doing some tests with lz4 trunk and I am having trouble with
uncompress_unknownOutputSize and compressHC. More specifically, I am testing it
on random char streams (same likelyhood for every char in the [0, 256[ range),
then compress with compressHC and uncompress_unknownOutputSize always returns
an uncompressed size of 4096 when the size of the original stream size is ~4200
or more.
On the other hand, it works perfectly fine:
- when the original size of the stream is < 4000,
- when I decompress with uncompress, whatever the original stream size is,
- with a non-random stream, whatever the original stream size is.
Original issue reported on code.google.com by [email protected]
on 18 Jul 2012 at 4:28
This is a first attempt at limited output for LZ4HC. Passes fuzzer tests, slows
compression by <3% on i7.
Original issue reported on code.google.com by [email protected]
on 24 Aug 2012 at 8:49
Attachments:
What steps will reproduce the problem?
1. LZ4_uncompress returns the number of input bytes processed
What is the expected output? What do you see instead?
I would expect it to return the number of bytes decompressed. I already know
the size of my input and that is what always shows in the return value.
What version of the product are you using? On what operating system?
Using on iOS 6, iPad 3
Please provide any additional information below.
I've modified the return value to be this:
return 5 + (int)((char*)op - dest); // return the num bytes output
For my purposes this returns the number of bytes uncompressed, although I have
to add 5 because the op pointer doesn't seem to be at the real end when the
algorithm finishes.
Original issue reported on code.google.com by [email protected]
on 27 Sep 2012 at 4:03
When compiling lz4 on NetBSD, the build fails because of the obsolete ftime()
function. It still does exist on NetBSD, but you would have to link against
-lcompat.
Even on Linux, the ftime man page says:
"This function is obsolete. Don't use it." and recommends gettimeofday()
instead, which is provided by BSDs since 4.2BSD and has been added to the POSIX
standard in 2001.
The attached patch replaces ftime() with over to gettimeofday().
Original issue reported on code.google.com by [email protected]
on 23 Oct 2012 at 9:04
Attachments:
That's not the duplicate of the previous issue. I've changed the data being
compressed and this happened again.
1. Run the stress test (Windows) http://directnet-drive.net/lz4_test.cpp (I've
uploaded a new version of the test)
2. Program crashes
3. #define LZ4_WORKAROUND to workaround the bug
lz4 latest svn source (r25), compiled using VS2010, run on Windows 7
lz crashes on the lines
do { *(U32*)op = *(U32*)ip; op+=4; ip+=4; } while (op<ref) ;
and
ip++; forwardH = HASH_VALUE(ip);
Original issue reported on code.google.com by [email protected]
on 21 Sep 2011 at 2:24
What steps will reproduce the problem?
1. Compress data.
2. Recompress the same input with output limited to exactly the length of the
previous run.
What is the expected output? What do you see instead?
Compression sometimes fails when there is exactly enough space.
What version of the product are you using? On what operating system?
svn r77, Arch Linux x86_64
Please provide any additional information below.
The calculation of the encoded run length was not exact, and could differ from
the actual run length by a small amount in some cases. op + (encoded length)
was also being compared as greater than or equal to oend, but a result exactly
equal to oend means that the encoded data will exactly fill the buffer. I
believe the attached patch corrects all length tests in lz4.c properly. lz4demo
-b shows a small cost in compression speed, about 2.5% for larger data and
about 7% for the 64k algorithm.
Original issue reported on code.google.com by [email protected]
on 24 Aug 2012 at 4:00
Attachments:
1. Run the stress test (Windows) http://directnet-drive.net/lz4_test.cpp
2. Program crashes
3. #define LZ4_WORKAROUND to workaround the bug
lz4 latest svn source, compiled using VS2010, run on Windows 7
lz crashes on the line
for(;length>0;length-=4) { *(U32*)op = *(U32*)anchor; op+=4; anchor+=4; }
It tries to access 4 bytes at anchor address but it points to the end of the
input stream minus 1/2/3 bytes.
Original issue reported on code.google.com by [email protected]
on 14 Sep 2011 at 12:04
What steps will reproduce the problem?
1. make clean
2. make
What is the expected output? What do you see instead?
lz4demo.c: In function ‘decode_file’:
lz4demo.c:294:9: warning: ignoring return value of ‘fwrite’, declared with
attribute warn_unused_result
lz4demo.c: In function ‘compress_file’:
lz4demo.c:202:8: warning: ignoring return value of ‘fwrite’, declared with
attribute warn_unused_result
lz4demo.c:223:9: warning: ignoring return value of ‘fwrite’, declared with
attribute warn_unused_result
What version of the product are you using? On what operating system?
gcc-4.5.3, glibc-2.14.1-r3, linux
Please provide any additional information below.
return value of fwrite is not checked anywhere, at all.
which means compression or decompression will silently fail on a full disk
also output may be corrupted due to temporary disk error, e.g. comp/decomp
output to a network file system
Original issue reported on code.google.com by [email protected]
on 29 Nov 2012 at 12:28
What steps will reproduce the problem?
1. Read attached file as byte array
2. Compress it
3. Decompress Unknown output size
What is the expected output? What do you see instead?
Decompressed output should match input.
The decompressed array is too short.
What version of the product are you using? On what operating system?
I am actually using the LZ4Sharp port but changing line 850-ish from
if (op == oend) break; // Check EOF (should never happen, since last 5 bytes
are supposed to be literals)
to
if (op == oend) goto _output_error;
fixes the problem because then whatever is looping and increasing the output
array size knows to run it again with a bigger size. The comment also indicates
this is an error condition.
Please provide any additional information below.
The data file is attached.
The error may only show up when increasing the output array in certain size
increments. I was using input array * 4 to start and multiplying by 4 each time
through the loop.
Original issue reported on code.google.com by [email protected]
on 2 Oct 2012 at 8:39
Attachments:
What steps will reproduce the problem?
1. when extracting compressed data
2.
3.
What is the expected output? What do you see instead?
LZ4_uncompress output differ
What version of the product are you using? On what operating system?
linuxmint maya x64, kernel 3.4.4
Please provide any additional information below.
lz4 rev68, compressed with lz4hc, some bytes differ when decompressing
compressed size 638MB, uncompressed size around 1.8GB
different 1
extracted_file1:
03EE DCE0: 32 42 39 44 41 38 44 38 39 45 44 39 30 39 34 35 2B9DA8D8 9ED90945
extracted_file2:
03EE DCE0: 32 42 31 44 41 38 44 38 39 45 44 39 30 39 34 35 2B1DA8D8 9ED90945
different 2
extracted_file1:
3D36 DF20: CD C6 B2 FF CE C8 BB FF D0 CA BE FF D2 CC C1 FF ........ ........
extracted_file2:
3D36 DF20: CD C6 BA FF CE C8 BB FF D0 CA BE FF D2 CC C1 FF ........ ........
different 3
extracted_file1:
3EA7 6F20: A1 D7 17 BC 06 6A 9C 11 2C 4F 31 D6 16 64 74 B4 .....j.. ,O1..dt.
extracted_file2:
3EA7 6F20: A1 D7 13 BC 06 6A 9C 11 2C 4F 31 D6 16 64 74 B4 .....j.. ,O1..dt.
different 4
extracted_file1:
528F 9F60: 83 78 10 00 74 12 8B 45 FC 8B 40 18 8B 4D FC 8B .x..t..E [email protected]..
extracted_file2:
528F 9F60: 83 78 18 00 74 12 8B 45 FC 8B 40 18 8B 4D FC 8B .x..t..E [email protected]..
some different more, and they looks like randomly, but no idea why.
regards,
Mark
Original issue reported on code.google.com by [email protected]
on 23 Jun 2012 at 2:17
Hi Yann,
If you give LZ4 HC a sequence such as
[1, 2, 3, 4, 5, 6, [...65530 incompressible bytes...], 1, 2, 3, 4, 5, 6, [other
bytes]]
LZ4 HC won't be able to compress it because it always cheks ref >
ip-MAX_DISTANCE instead of ref >= ip-MAX_DISTANCE (see attached patch for more
details).
Thank you!
Original issue reported on code.google.com by [email protected]
on 29 Nov 2012 at 10:58
Attachments:
I received a failure report [1] from the CPAN testers in which __builtin_ctz
was an undefined symbol. A bit of searching shows that was added in GCC 3.4 and
the tester's environment had version 3.3.3. I suggest testing for the version
of GCC and providing a fallback function if the builtin isn't available in the
user's version.
[1] http://www.cpantesters.org/cpan/report/7b034a9c-581e-11e1-a156-f932921a7b64
Original issue reported on code.google.com by [email protected]
on 16 Feb 2012 at 9:02
Functions LZ4_uncompress() and LZ4_uncompress_unknownOutputSize() use restrict
keyword with local variables op and ref:
const BYTE* restrict ref;
BYTE* restrict op = (BYTE*) dest;
As far as I understand how 'restrict' works, this seems like an invalid use
that can fool the compiler.
In fact, when using icc version 11.0 (with plenty of optimizations on), this
results in wrong decompression output (fixed by getting rid of restrict on both
variables).
Original issue reported on code.google.com by [email protected]
on 24 Sep 2012 at 2:20
I have compiled the latest code using Visual Studio 2005 and the generic CE 6.0
SDK, as well as Pocket PC 2003 and a specific handheld device, all ARM4 based.
_FORCE_SW_BITCOUNT is defined (though generates the warning "unary minus
operator applied to unsigned type, result still unsigned".)
The LZ4_uncompress function throws an exception at LZ4_WILDCOPY (line 657 of
lz4.c.)
Original issue reported on code.google.com by [email protected]
on 17 Feb 2012 at 1:07
A user reported invalid roundtrip results for the Perl interface under Ubuntu
amd64. I was able to able to reproduce using Virtualbox with the plain C
library. The error is not present when compiling as a 32-bit executable with
-m32.
Attached are the c program I used and the input file.
$ uname -a
Linux ubuntu 3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64
x86_64 x86_64 GNU/Linux
Original issue reported on code.google.com by [email protected]
on 17 Mar 2012 at 12:39
1. Compile lz4 for ARM cpu
2. Turn on SIGBUS signals generation in case of unaligned access (on linux can
be done via "echo 5 > /proc/cpu/alignment")
3. Run lz4 test
I'm trying it on linux 2.4.21.
The core of the problem is that ARM/MIPS cpu does not allow unaligned access,
so that the following code crashes:
char a[8];
*(int*) &a[1] = 0; // crash because of unaligned write to mem
int b = *(int*) &a[1]; // crash because of unaligned read from mem
In lz4 there is a plenty of such lines:
do { *(U32*)op = *(U32*)anchor; op+=4; anchor+=4; } while (op<l_end) ;
in case of op pointer not aligned to 32-bit boundary that would cause crash
(SIGBUS).
The possible solution could be using __attribute__ ((packed)) if compiling
using gcc (that's almost always true for ARM/MIPS).
In case of x86 & x64 that won't affect performance at all, the generated code
should remain the same. In case of ARM/MIPS that forces compiler to generate
per-byte access code. Plz take a look at the patch:
http://directnet-drive.net/lz4.20.patch
(result here: http://directnet-drive.net/lz4.c)
That fixes the problem in my case and ARM tests succeed. x86 and x64 code is
not affected at all.
Although ARM/MIPS cpus are not so widespread now they are becoming more and
more popular. If you need any kind of my help please do not hesitate to ask for
it.
Original issue reported on code.google.com by [email protected]
on 21 Sep 2011 at 3:16
According to the msdn page:
http://msdn.microsoft.com/en-us/library/ms253923%28v=vs.80%29.aspx
VC device compiler are not support bit-scan intrinsics.
So LZ4_FORCE_SW_BITCOUNT should be defined if _MSC_VER and _WIN32_WCE are both
defined.
Original issue reported on code.google.com by [email protected]
on 12 Apr 2012 at 9:47
What steps will reproduce the problem?
1. Take 5 gb logfile
2. Compress with LZ4
3. Decompress
What is the expected output? What do you see instead?
Expected to see original file, got a 0byte file
What version of the product are you using? On what operating system?
LZ4 official windows binary, v1.3, on win7, win server2008r2
Please provide any additional information below.
Files 1 to 1.5 gb work perfeclty fine.
Original issue reported on code.google.com by [email protected]
on 26 Aug 2012 at 10:11
STEPS:
1. Create a Win32 console project
2. Add LZ4 source files to the project
3. Start build
EXPECTED RESULT:
Successfull build
ACTUAL RESULT:
Build failed with error:
lz4hc.obj : error LNK2019: unresolved external symbol __BitScanForward
referenced in function _LZ4_NbCommonBytes
Original issue reported on code.google.com by specialforest
on 29 Jun 2012 at 2:32
On gcc and clang, we can use:
#define forceinline __inline__ __attribute__((always_inline))
#define ensure_forceinline __attribute__((always_inline)) // inline or die
Original issue reported on code.google.com by [email protected]
on 12 Apr 2012 at 10:03
The Free Software Foundation is no longer at 675 Mass Ave, Cambridge, MA.
Our new address is 51 Franklin Street, Suite 500, Boston, MA 02110.
You can confirm this for yourself here: http://www.fsf.org/about/contact/
Please update all references to our old address in your code so people can
continue to contact us (we haven't been at the old address for more than a
decade at this point)
Kind regards,
matt
--
Matt Lee
Campaigns Manager
Free Software Foundation
Original issue reported on code.google.com by [email protected]
on 8 Sep 2011 at 3:35
Hello,
It appears that there is an issue with the compression routine reading past the
end of the array for certain inputs.
Please see the attached fuzz test. It will crash on the second iteration.
Original issue reported on code.google.com by [email protected]
on 29 Feb 2012 at 1:26
Attachments:
With Visual Studio 2012, changing "BYTE token" to "unsigned token" in
"LZ4_uncompress" removes one byte=>dword instruction and makes the decompressor
approx 2.5% faster in my benchmark.
Original issue reported on code.google.com by [email protected]
on 5 Dec 2012 at 6:51
What steps will reproduce the problem?
1. Download code, gmake
What is the expected output? What do you see instead?
Successful compile.
In file included from bench.c:50:0:
/usr/include/stdio.h:333:29: error: unknown type name 'off64_t'
/usr/include/stdio.h:334:1: error: unknown type name 'off64_t'
Fixed by removing the #define _LARGEFILE64_SOURCE line (line 33)
Could maybe replace with
#ifndef __LP64__
#ifdef __sun__
#define _LARGEFILE_SOURCE
#define FILE_OFFSET_BITS=64
#else
#define _LARGEFILE64_SOURCE
#endif
#endif
Or better move it into the makefile with with a `getconf LFS_CFLAGS` on the
compile line.
Original issue reported on code.google.com by [email protected]
on 7 Oct 2012 at 2:18
What steps will reproduce the problem?
1. call LZ4_uncompress_unknownOutputSize with an osize that is exactly the size
of the uncompressed data. It will return a decompressed size that is too small.
What is the expected output? I expected 65535 What do you see instead? I got
65531
The bug is in line 534. The line following a memcpy. ip += length; Since
LZ4_uncompress_unknownOutputSize uses op to calculate the length unless there
is an error this line should be op += length;
Original issue reported on code.google.com by [email protected]
on 26 Oct 2011 at 7:05
In some situation, LZ4_compress_limitedOutput() may write beyond provided
output buffer.
Original issue reported on code.google.com by [email protected]
on 21 Aug 2012 at 4:27
As part of testing out LZ4 I have compiled and run it on a
collection of machines on our build cluster including some with
very old compilers.
I needed to make a number of changes in order for lz4.c to
compile and work everywhere. I did not test lz4hc.c, but it
probably needs a matching set of changes.
The build cluster includes:
x86 linux on Debian: 32 & 64bit versions of all releases since 3.0
x86 freebsd: all 32 & 64bit versions since 6.0
netbsd 1.6.1 on 32bit x86
netbsd 4.0 on 32/64bit x86
OpenBSD 4.2 on x86
linux on ia64 & PowerPC
Solaris on x86 and Sparc
SCO OpenServer R5 on x86
MacOS 10.3 PowerPC
MacOS 10.4 x86
AIX 5 on PowerPC
HP-UX on PARISC and IA-64
and probably some other oddballs that I forgot about
Here were the changes I needed to make
64-bit detection:
The define for 64-bit code was missing __ia64__
BIG_ENDIAN:
For big endian I added __hpux__ && __sparc__
I didn't test this part, but it would probably be a good idea to
also add a test of:
&& !(defined(__BYTE_ORDER__) &&
(__BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__))
The powerpc can actually be either, but in most common cases
it is big endian so that part of the test seems to work.
Packed structures:
The '#pragma pack(1)' directive for the U32_S structures doesn't
work in older GCCs or with the ia64 architecture.
In my searching I found the __packed__ attribute was more portable,
so I switched to using that instead. Not sure if that will
work with MSVC or not as that is not one of my targets.
stdint.h
The stdint.h is too new to be included on all of these machines so
used typedefs directly. (why #defines before?)
In theory these defines can be wrong, but in practice it works on
every architecture in common use.
Original issue reported on code.google.com by [email protected]
on 5 Oct 2012 at 2:26
Attachments:
> uname -v
Darwin Kernel Version 11.4.0: Mon Apr 9 19:33:05 PDT 2012;
root:xnu-1699.26.8~1/RELEASE_I386
LZ4 stopped building after revision r71:
> svn checkout -q http://lz4.googlecode.com/svn/trunk/ lz4-trunk && cd
lz4-trunk && make
gcc -O3 -I. -std=c99 -Wall -W -Wundef -Wno-implicit-function-declaration
lz4hc.c lz4.c bench.c lz4demo.c -o LZ4Demo.exe
Undefined symbols for architecture x86_64:
"_LZ4_compress64kCtx", referenced from:
_LZ4_compress_limitedOutput in ccJ02lMQ.o
_LZ4_compress in ccJ02lMQ.o
"_LZ4_compressCtx", referenced from:
_LZ4_compress_limitedOutput in ccJ02lMQ.o
_LZ4_compress in ccJ02lMQ.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make: *** [lz4demo] Error 1
Revisions before r71 build ok:
> svn checkout -q -r 70 http://lz4.googlecode.com/svn/trunk/ lz4-ok && cd
lz4-ok && make
gcc -O3 -I. -std=c99 -Wall -W -Wundef -Wno-implicit-function-declaration
lz4hc.c lz4.c bench.c lz4demo.c -o LZ4Demo.exe
Original issue reported on code.google.com by [email protected]
on 4 Aug 2012 at 9:08
With Visual Studio 2012, changing:
*op++ = *ref++;
*op++ = *ref++;
*op++ = *ref++;
*op++ = *ref++;
into:
op[0] = ref[0];
op[1] = ref[1];
op[2] = ref[2];
op[3] = ref[3];
op += 4, ref += 4;
gives a 1.5% speed boost in my benchmark.
Original issue reported on code.google.com by [email protected]
on 5 Dec 2012 at 7:21
What steps will reproduce the problem?
1. cmake cmake
2. make
What is the expected output? What do you see instead?
Expected a working build. Got:
rhino ➜ ~/projects/lz4-read-only make
[ 16%] Building C object CMakeFiles/lz4demo32.dir/lz4.c.o
[ 33%] Building C object CMakeFiles/lz4demo32.dir/bench.c.o
[ 50%] Building C object CMakeFiles/lz4demo32.dir/lz4demo.c.o
Linking C executable lz4demo32
CMakeFiles/lz4demo32.dir/lz4demo.c.o: In function `compress_file':
lz4demo.c:(.text+0x25c): undefined reference to `LZ4_compressHC'
CMakeFiles/lz4demo32.dir/bench.c.o: In function `BMK_benchFile':
bench.c:(.text+0x138): undefined reference to `LZ4_compressHC'
collect2: ld returned 1 exit status
make[2]: *** [lz4demo32] Error 1
make[1]: *** [CMakeFiles/lz4demo32.dir/all] Error 2
make: *** [all] Error 2
What version of the product are you using? On what operating system?
subversion checkout.
Please provide any additional information below.
When I changed CMakeLists.txt from
set(LZ4_SRCS_LIB ${SRC_DIR}lz4.c ${SRC_DIR}lz4.h )
to
set(LZ4_SRCS_LIB ${SRC_DIR}lz4.c ${SRC_DIR}lz4hc.c ${SRC_DIR}lz4.h )
the build proceeded correctly.
Original issue reported on code.google.com by [email protected]
on 6 Jun 2012 at 12:11
Hi,
The header for LZ4_Decompress() states:
// Note : The decoding functions LZ4_uncompress() and
LZ4_uncompress_unknownOutputSize()
// are safe against "buffer overflow" attack type.
// They will never write nor read outside of the provided input
and output buffers.
// A corrupted input will produce an error result, a negative int,
indicating the position of the error within input stream.
However, the following demo program shows that this is not always the case:
#include <stdio.h>
#include <string.h>
#include "lz4.h"
int main(int argc, char** argv)
{
char *input, *output;
input = (char *)malloc(3);
input[0] = 0xf0;
input[1] = 0xfe;
input[2] = 0x00;
output = (char *)malloc(4096);
int ret = LZ4_uncompress_unknownOutputSize(input, output, 3, 4096);
printf("%d\n", ret);
return 0;
}
Running it with Valgrind reveals:
==26795== Invalid read of size 8
==26795== at 0x400750: LZ4_uncompress_unknownOutputSize
(/home/sesse/lz4-read-only/lz4.c:744)
==26795== by 0x400667: main (/home/sesse/lz4-read-only/democase.c:16)
==26795== Address 0x51b6072 is 2 bytes inside a block of size 3 alloc'd
==26795== at 0x4C2A4D6: malloc
(/home/samsonov/valgrind-variant/valgrind-test/coregrind/m_replacemalloc/vg_repl
ace_malloc.c:263)
==26795== by 0x40063A: main (/home/sesse/lz4-read-only/democase.c:9)
==26795==
==26795== Invalid read of size 8
==26795== at 0x400761: LZ4_uncompress_unknownOutputSize
(/home/sesse/lz4-read-only/lz4.c:744)
==26795== by 0x400667: main (/home/sesse/lz4-read-only/democase.c:16)
==26795== Address 0x51b607a is 7 bytes after a block of size 3 alloc'd
==26795== at 0x4C2A4D6: malloc
(/home/samsonov/valgrind-variant/valgrind-test/coregrind/m_replacemalloc/vg_repl
ace_malloc.c:263)
==26795== by 0x40063A: main (/home/sesse/lz4-read-only/democase.c:9)
This is with LZ4 as of r57, 64-bit x86, GCC 4.4.3.
Original issue reported on code.google.com by [email protected]
on 1 Mar 2012 at 11:34
Hi Yann,
If you give LZ4 HC a sequence such as
[1, 2, 3, 4, 5, 6, [...65530 incompressible bytes...], 1, 2, 3, 4, 5, 6, [other
bytes]]
LZ4 HC won't be able to compress it because it always cheks ref >
ip-MAX_DISTANCE instead of ref >= ip-MAX_DISTANCE (see attached patch for more
details).
Thank you!
Original issue reported on code.google.com by [email protected]
on 29 Nov 2012 at 10:58
Attachments:
Currently the users have to estimate the worst-case compressed size by
following the guidance in the comment to LZ4_compress; moreover, it's not
possible to use a smaller buffer (i.e. we expect the data to be at least 50%
compressible, so we want to save memory in the common case).
It would be nice to have:
1. A function for converting input size to worst-case output size (i.e. see
compressBound() in zlib)
2. A function for safe compression with a specified output buffer size that's
guaranteed to never write past the end; return negative number or zero to
indicate error
3. (related) A function for safe decompression with a specified input buffer
size that's guaranteed to never read past the end of input buffer.
Original issue reported on code.google.com by [email protected]
on 11 Feb 2012 at 7:06
You need to add:
#pragma intrinsic(_BitScanForward)
Otherwise VC will complain an error (unresolved symbol) at link time of DEBUG
Version.
This is because the "/Oi" option are not specified while compiling the debug
version.
Other bit-scan intrinsics are similar.
Original issue reported on code.google.com by [email protected]
on 12 Apr 2012 at 8:52
LZ4 library should be refactored to support streaming compression and
decompression similar to how zlib works.
There would be an lz4Init(), lz4(), and lz4End() similar to how the inflate
functions work in zlib. You would pass around a pointer to the stream for each
function. Call to lz4() would save the state of the compression/decompression
machine for subsequent calls to lz4(). Here is zlib's stream structure.
typedef struct z_stream_s {
z_const Bytef *next_in; /* next input byte */
uInt avail_in; /* number of bytes available at next_in */
uLong total_in; /* total number of input bytes read so far */
Bytef *next_out; /* next output byte should be put there */
uInt avail_out; /* remaining free space at next_out */
uLong total_out; /* total number of bytes output so far */
z_const char *msg; /* last error message, NULL if no error */
struct internal_state FAR *state; /* not visible by applications */
alloc_func zalloc; /* used to allocate the internal state */
free_func zfree; /* used to free the internal state */
voidpf opaque; /* private data object passed to zalloc and zfree */
int data_type; /* best guess about the data type: binary or text */
uLong adler; /* adler32 value of the uncompressed data */
uLong reserved; /* reserved for future use */
} z_stream;
Original issue reported on code.google.com by [email protected]
on 4 Nov 2012 at 9:40
What steps will reproduce the problem?
1. gmake
2. run ./lz4demo.exe ./lz4demo.exe /dev/null
What is the expected output? What do you see instead?
Successful compress. Bus error. Core dump.
What version of the product are you using? On what operating system?
Solaris 10. sparc.
Please provide any additional information below.
When compiling with SolarisStudio I am forced to compile with -xmemalgin=1i
anyway, and the problem does not appear. Guess it is a performance killer.
I have not found a similar option for gcc.
When I changed some *(U32*)x to A32(x) it worked .
212c211
< #define HASH_VALUE(p) HASH_FUNCTION(A32(p))
---
> #define HASH_VALUE(p) HASH_FUNCTION(*(U32*)(p))
345c344
< if (A32(ref) == A32(ip))
---
> if (*(U32*)ref == *(U32*)ip)
388c387
< if (A32(ref) == A32(ip))
---
> if (*(U32*)ref == *(U32*)ip)
Original issue reported on code.google.com by [email protected]
on 30 Nov 2012 at 10:51
The comments for LZ4_uncompress() in lz4.h state:
"This function never writes beyond dest + osize, and is therefore protected
against malicious data packets"
I don't see how can this be true, if it doesn't even take isize as a parameter.
Surely a malicious attacker could truncate the input data, causing reads beyond
the end of the input buffer for any given expected osize, not to mention more
sophisticated attacks "fishing" data past the end of the input buffer.
Is there something I'm missing?
I know there's LZ4_uncompress_unknownOutputSize for safe decoding, but if I'm
right I think this should be clarified.
Original issue reported on code.google.com by [email protected]
on 31 Oct 2012 at 2:12
On NetBSD, including stdlib.h pulls in bswap.h which defines bswap16.
Reference error:
http://www.cpantesters.org/cpan/report/34d82f1a-71a5-11e1-8755-e63c2a028b78
You might be able to borrow some of the portability tweaks from csnappy:
https://github.com/zeevt/csnappy/blob/master/csnappy_internal_userspace.h
Original issue reported on code.google.com by [email protected]
on 19 Mar 2012 at 6:09
What steps will reproduce the problem?
1. Use ubuntu 8.04 with gcc (gcc version 4.2.4 (Ubuntu 4.2.4-1ubuntu4))
2. Just run make
I use sources from svn on ubuntu 8.04. I use gcc 4.2.4, but that don't have
__builtin_bswap32. And i have errors after run make.
gcc -O3 -I. -std=c99 -Wall -W -Wundef -Wno-implicit-function-declaration
lz4hc.c lz4.c bench.c lz4demo.c -o lz4demo64
/tmp/ccqSPzan.o: In function `decode_file':
lz4demo.c:(.text+0x3dd): undefined reference to `__builtin_bswap32'
lz4demo.c:(.text+0x468): undefined reference to `__builtin_bswap32'
/tmp/ccqSPzan.o: In function `compress_file':
lz4demo.c:(.text+0x756): undefined reference to `__builtin_bswap32'
lz4demo.c:(.text+0x771): undefined reference to `__builtin_bswap32'
lz4demo.c:(.text+0x7ae): undefined reference to `__builtin_bswap32'
collect2: ld returned 1 exit status
make: *** [lz4demo64] Error 1
I have patched your sources for fix that problem.
--- lz4demo.c (revision 67)
+++ lz4demo.c (working copy)
@@ -58,8 +58,13 @@
#if defined(_MSC_VER) // Visual Studio
#define swap32 _byteswap_ulong
#elif GCC_VERSION >= 402
+#include <byteswap.h>
+#if defined(__builtin_bswap32)
#define swap32 __builtin_bswap32
#else
+#define swap32 __bswap_32
+#endif
+#else
Original issue reported on code.google.com by [email protected]
on 9 Jun 2012 at 8:10
What steps will reproduce the problem?
1. Compile with
cc -xc99 -fast -I. -m32 lz4.c lz4hc.c bench.c lz4demo.c -o lz4demo.exe
-xmemalign=1i -erroff=E_WHITE_SPACE_IN_DIRECTIVE
2. run ./lz4demo.exe ./lz4demo.exe target && ./lz4demo.exe -d target back &&
cmp ./lz4demo.exe back
What is the expected output? What do you see instead?
No errors from cmp.
Instead:
./lz4demo.exe back differ: char 44, line 1
What version of the product are you using? On what operating system?
Solaris 10 sparc. cc: Sun C 5.12 SunOS_sparc Spica 2011/07/21
Please provide any additional information below.
This can be fixed by moving the #include directives to the top.
Solaris Studio needs to include some files before it can have sensible values
for BIG_ENDIAN or BYTE_ORDER.
I know I have reported this before and not followed up.
This time I promise to follow up and do all tests you ask of me.
Original issue reported on code.google.com by [email protected]
on 26 Nov 2012 at 2:52
What steps will reproduce the problem?
1. No lz4.so library available
What is the expected output? What do you see instead?
More of a feature request, I suppose. I'd fork it and make a pull request if
you were on github:) I'd like to be able to link to lz4.so from Haskell.
Original issue reported on code.google.com by [email protected]
on 6 Jun 2012 at 9:36
What steps will reproduce the problem?
1. char * c = "abcd";
2. c[2] = '\0';
3. compress c & fail.
I'd like to be able to compress arbitrary data.
https://github.com/mwotton/lz4hs has more detailed test cases.
Original issue reported on code.google.com by [email protected]
on 6 Jun 2012 at 11:13
What steps will reproduce the problem?
1. svn checkout http://lz4.googlecode.com/svn/trunk/ lz4-read-only
Actual result:
svn: Server sent unexpected return value (307 Temporary Redirect) in response
to OPTIONS request for 'http://lz4.googlecode.com/svn/trunk'
What is the expected output? What do you see instead?
Get the cource code.
What version of the product are you using? On what operating system?
tigervnc-1.0.90-0.17.20110314svn4359.el6.x86_64
Original issue reported on code.google.com by [email protected]
on 11 Jul 2012 at 2:27
What steps will reproduce the problem?
1. svn co http://lz4.googlecode.com/svn/
2. cd svn/trunc
3. gmake CC="gcc -m64" lz4demo
What is the expected output? What do you see instead?
Succesfully built 64 bit lz4demo.
Instead I get :
In file included from bench.c:56:0:
/usr/include/stdio.h:333:29: error: unknown type name 'off64_t'
/usr/include/stdio.h:334:1: error: unknown type name 'off64_t'
gmake: *** [lz4demo] Error 1
What version of the product are you using? On what operating system?
Latest. Solaris 10 on sparc.
Please provide any additional information below.
The fix is to change
#if (defined(__sun__) && (!defined(__LP64__))) // Sun Solaris 32-bits
requires specific definitions
# define _LARGEFILE_SOURCE
# define FILE_OFFSET_BITS=64
#else
# define _LARGEFILE64_SOURCE
#endif
to
#if (defined(__sun__) && (!defined(__LP64__))) // Sun Solaris 32-bits
requires specific definitions
# define _LARGEFILE_SOURCE
# define FILE_OFFSET_BITS=64
#elif ! defined(__LP64__) // No point defining Large file for 64 bit
# define _LARGEFILE64_SOURCE
#endif
Original issue reported on code.google.com by [email protected]
on 26 Nov 2012 at 2:31
What steps will reproduce the problem?
1. make all
Actual result:
make all
gcc -m32 -Os -march=native -I. -std=c99 -Wall -W -Wundef
-Wno-implicit-function-declaration lz4hc.c lz4.c bench.c lz4demo.c -o lz4demo32
In file included from /usr/include/features.h:385,
from /usr/include/stdlib.h:25,
from lz4hc.c:83:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
In file included from /usr/include/features.h:385,
from /usr/include/stdlib.h:25,
from lz4.c:139:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
In file included from /usr/include/features.h:385,
from /usr/include/stdlib.h:25,
from bench.c:48:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
In file included from /usr/include/features.h:385,
from /usr/include/stdio.h:28,
from lz4demo.c:40:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
make: *** [lz4demo32] Error 1
What is the expected output? What do you see instead?
Binaries created
What version of the product are you using? On what operating system?
libgcc-4.4.6-4.el6.x86_64
gcc-4.4.6-4.el6.x86_64
libgcc-4.4.6-4.el6.i686
Original issue reported on code.google.com by [email protected]
on 11 Jul 2012 at 3:26
I'm attaching a patch which properly declares source arrays as const.
This is needed to prevent warnings or errors when passing const source data to
LZ4 methods.
James
Original issue reported on code.google.com by [email protected]
on 30 Jan 2012 at 8:11
Attachments:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.