Add optional FreeARC compression to pak files about urho3d

[cadaver]

As an initial look the library does look a bit nasty

• No ASCII / English comments for the most part
• No license defined
• It is unclear how to read uncompressed data from a compressed file as a stream

To be honest I had not heard of FreeARC before you mentioned it. Though it may be technically very good, using something like zziplib instead would guarantee a well-defined license, and I know its API is sufficient for the use required by Urho3D's archives.

[Canardian]

Yeah, I thought it was a bit unclear how the license goes, or how well it supports to be used as a library. The author only says on the download page: "FreeArc is free for both commercial and non-commercial users.".

As alternative decompression library I would suggest qpress:

qpress is a portable file archiver using QuickLZ and designed to utilize fast storage systems to their max. It's often faster than file copy because the destination is smaller than the source. A few features:

• multiple cores, reaching upto 1.1 Gbyte/s in-memory compression on a quad core i7
• 64-bit file sizes and tested with terabyte sized archives containing millions of files and directories
• pipes and redirection and *nix-like behaviour for scripting and flexibility
• Adler32 checksums to ensure that decompressed data has not been corrupted
• data recovery of damaged archives with 64 Kbyte grannularity
• unbuffered disk I/O (Windows only) to prevent disk cache of other applications from being flushed

http://www.quicklz.com

However, I'm not sure if the license allows commercial games made with Urho3D. I guess not, if GPL is inherited from engine to game?

// QuickLZ can be used for free under the GPL-1 or GPL-2 license (where anything 
// released into public must be open source) or under a commercial license if such 
// has been acquired (see http://www.quicklz.com/order.html). The commercial license 
// does not cover derived or ported versions created by third parties under GPL.

So the next suggestion would be xz, which is now also part of Debian, so it must be good :)
http://tukaani.org/xz/

The most interesting parts of XZ Utils (e.g. liblzma) are in the public domain. You can do whatever you want with the public domain parts.
Some parts of XZ Utils (e.g. build system and some utilities) are under different free software licenses such as GNU LGPLv2.1, GNU GPLv2, or GNU GPLv3.

[cadaver]

Yes, we cannot take libraries with licenses of the GPL or LGPL variety, as we must be able to link everything together statically while not contaminating the engine with a license that would require the games themselves to be open sourced.

[szamq]

If you are looking for compression library, I can recommend snappy from Google
https://code.google.com/p/snappy/
It's fast, really easy to use, multi-platform and on BSD license.

[Canardian]

While comparing liblzma (=xz) and snappy (I'm not very keen to use zip, since it's a commercial format, and rather a hack), I found also another interesting library called lz4 (also BSD license), which is said to be much faster than snappy: https://code.google.com/p/lz4/

I'm continuing to compare xz and lz4 and see which fits better, although I really like the high speed of lz4 while its compression is also better than snappy's. I assume xz (which uses LZMA2) packs a lot better than lz4, but lz4 is much faster (may even exceed uncompressed speed, because there is less IO to do). On the other hand lz4 has also a maximum compression mode, which makes it probably close to xz ratio.

[Canardian]

Ok, I am convinced! lz4 supports also streaming and is way faster than even FreeARC, and naturally faster than xz (=7-zip LZMA2):
http://pokecraft.first-world.info/wiki/Quick_Benchmark:_Gzip_vs_Bzip2_vs_LZMA_vs_XZ_vs_LZ4_vs_LZO
http://fastcompression.blogspot.fi/p/compression-benchmark.html

[cadaver]

Can you describe how you would do streaming decompression with lz4? From the header file it wasn't entirely obvious. Would you allocate a bigger buffer into which you'd decompress data in bigger blocks and then read off that?

[Canardian]

The author of lz4 says one should use the LZ4_compressBound() function to get a good estimation of the output buffer size:
https://code.google.com/p/lz4/issues/detail?id=8

[boberfly]

I'd recommend lzham:
https://code.google.com/p/lzham/

This would also benefit if crunch is used to compress textures, which I plan to use... :)

Having lzham being used in the wild and it being very optimal in a game runtime environment and a zlib API, there's too many wins here.

[Canardian]

It seems lzham is much slower than lz4 (and this confirms once again that lz4 is superior in speed):
http://cbloomrants.blogspot.fi/2012/10/10-02-12-small-note-on-lzham.html

[boberfly]

If speed is a lot more important, lz4 does look really good for decompression and dynamic compression for things like network packets perhaps. Just it is bigger in file size even against zlib. Lzham solves a different problem, allowing high compression ratios while still having good decompression speeds (albeit suffers currently with small files by the looks of it, unless the SVN branch fixes that). Also in lz4's issue tracker streaming doesn't seem to be supported?

[cadaver]

Integrating lz4 is quite painless (did a quick test); the lack of streaming does not matter as one can just pick a buffer size and use intermediate buffer streaming like is already being done for Android asset handling.

However it's true that lz4's compression ratios aren't that high, so I will test lzham next.

[Canardian]

At least for me the speed is primary need, since most big assets are packed anyway (maximum png compression, maximum ogg compression with VBR). I just want to have all assets neatly packed into one file and a bit encrypted so that it doesn't cause unnecessary security risks if there is clearly visible information in xml/text files. Also when launching the game, it should launch as fast as possible, so lz4 packed should be faster then even unpacked. And lz4 of course supports also high compression, which could be optionally enabled.

[cadaver]

LZ4 output is almost like plaintext until the matches initialize, so things like dialogue or shader code are somewhat clearly visible.

I will profile startup times of unpacked vs LZ4 vs lzham; the primary slowness of lzham is the first-time initialization of the dictionary but the same decompressor can be reused for different files which should mitigate this.

[Canardian]

I wouldn't leave zlib out of the race yet, it's still faster than lzham.

[cadaver]

Indeed, I was somehow under the impression that zlib was slower, while it clearly isn't.

[boberfly]

Something like 'crunch' optimises the dxt files to compress well with lzham/lzma after but that's a very specific use case (also the two projects crunch/lzham are by the same author).

I saw Battlefield 4 uses lz4 and Planetside 2 use lzham, not that that matters hehe. Hell Quake 3/Doom 3 used zlib it's still solid. :)

[cadaver]

The zlib API is nice, as you can decompress directly into the output buffer the caller has provided, and the files also get nicely scrambled, but reading files slowed down in the worst case about 6x from LZ4 or uncompressed.

I'll be merging LZ4 code soon.