dseeni / zippy

Pure Nim implementation of deflate, zlib, gzip and zip.

Zippy

nimble install zippy

Zippy is an implementation of DEFLATE, ZLIB and GZIP data compression formats. Zippy can also create and open Tarballs (.tar, .tar.gz, .tgz, .taz) and ZIP archives.

The goal of this library is to be a pure Nim implementation that is small, performant and dependency-free.

Zippy can also be used at compile time. This is great for baking assets into executables in compressed form. Check out an example here.

To ensure Zippy is compatible with other implementations, tests/validate.nim can be run. This script verifies that data compressed by Zippy can be uncompressed by other implementations (and that other implementations can uncompress data compressed by Zippy).

This library works well using Nim's relatively new --gc:arc and --gc:orc as well as the default garbage collector. This library also works using both nim c and nim cpp, in addition to --cc:vcc on Windows.

I have also verified that Zippy builds with --experimental:strictFuncs on Nim 1.4.0.

Examples

Simple examples using Zippy can be found in the examples/ folder.

• HTTP client gzip
• HTTP server gzip
• Compress a dir into a tarball or zip archive
• Extract from a tarball or zip archive
• Compose a tarball or zip archive in code

Performance

Benchmarks can be run comparing different deflate implementations. My benchmarking shows this library performs very well, a bit faster than zlib in some cases and a bit slower in others. Check the performance yourself by running tests/benchmark.nim.

nim c -d:release -r .\tests\benchmark.nim

The times below are measured on a Ryzen 5 5600X.

Compress

Each file is compressed 10 times per run.

Default compression

https://github.com/guzba/zippy results:

file	min	avg	std dv	runs	% red
alice29.txt	29.543 ms	30.698 ms	±0.188	x167	63.32%
urls.10K	157.261 ms	166.513 ms	±0.449	x31	67.49%
rfctest3.gold	5.486 ms	5.621 ms	±0.079	x973	70.73%
randtest3.gold	0.757 ms	0.777 ms	±0.015	x1000	0%
paper-100k.pdf	15.460 ms	16.046 ms	±0.109	x331	19.94%
geo.protodata	10.012 ms	10.434 ms	±0.068	x522	86.91%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file	min	avg	std dv	runs	% red
alice29.txt	58.013 ms	58.213 ms	±0.177	x86	64.23%
urls.10K	132.910 ms	133.379 ms	±0.380	x38	68.29%
rfctest3.gold	6.525 ms	6.589 ms	±0.060	x757	71.74%
randtest3.gold	0.887 ms	0.903 ms	±0.015	x1000	0%
paper-100k.pdf	15.020 ms	15.113 ms	±0.058	x331	20.59%
geo.protodata	8.996 ms	9.094 ms	±0.065	x549	87.24%

Fastest compression

https://github.com/guzba/zippy results:

file	min	avg	std dv	runs	% red
alice29.txt	12.745 ms	13.261 ms	±0.153	x405	55.32%
urls.10K	40.832 ms	41.720 ms	±0.131	x127	61.70%
rfctest3.gold	3.430 ms	3.522 ms	±0.022	x1000	66.31%
randtest3.gold	0.291 ms	0.306 ms	±0.009	x1000	0%
paper-100k.pdf	9.154 ms	9.399 ms	±0.049	x600	18.44%
geo.protodata	6.609 ms	6.747 ms	±0.016	x844	80.42%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file	min	avg	std dv	runs	% red
alice29.txt	12.797 ms	13.024 ms	±0.198	x384	57.17%
urls.10K	53.304 ms	53.515 ms	±0.122	x94	63.93%
rfctest3.gold	2.179 ms	2.220 ms	±0.029	x1000	67.53%
randtest3.gold	0.813 ms	0.831 ms	±0.016	x1000	0%
paper-100k.pdf	12.806 ms	12.878 ms	±0.056	x388	20.22%
geo.protodata	3.494 ms	3.531 ms	±0.024	x1000	84.12%

Best compression

https://github.com/guzba/zippy results:

file	min	avg	std dv	runs	% red
alice29.txt	36.293 ms	37.635 ms	±0.149	x136	63.75%
urls.10K	229.110 ms	244.351 ms	±0.431	x22	68.14%
rfctest3.gold	10.681 ms	11.173 ms	±0.121	x465	70.92%
randtest3.gold	0.762 ms	0.780 ms	±0.008	x1000	0%
paper-100k.pdf	16.124 ms	16.833 ms	±0.078	x311	20.07%
geo.protodata	10.795 ms	11.222 ms	±0.046	x459	87.07%

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file	min	avg	std dv	runs	% red
alice29.txt	86.051 ms	86.287 ms	±0.216	x58	64.38%
urls.10K	253.127 ms	253.952 ms	±0.724	x20	68.82%
rfctest3.gold	21.753 ms	22.018 ms	±0.168	x227	71.77%
randtest3.gold	0.887 ms	0.902 ms	±0.014	x1000	0%
paper-100k.pdf	16.749 ms	16.838 ms	±0.056	x297	20.64%
geo.protodata	12.252 ms	12.346 ms	±0.066	x405	87.37%

Uncompress

Each file is uncompressed 10 times per run.

https://github.com/guzba/zippy results:

file	min	avg	std dv	runs
alice29.txt.z	3.570 ms	3.595 ms	±0.018	x1000
urls.10K.z	17.324 ms	17.392 ms	±0.057	x288
rfctest3.z	0.777 ms	0.794 ms	±0.011	x1000
randtest3.z	0.067 ms	0.070 ms	±0.006	x1000
paper-100k.pdf.z	2.908 ms	2.937 ms	±0.034	x1000
geo.protodata.z	1.268 ms	1.299 ms	±0.016	x1000

https://github.com/nim-lang/zip results: (Requires zlib1.dll)

file	min	avg	std dv	runs
alice29.txt.z	3.530 ms	3.572 ms	±0.016	x1000
urls.10K.z	15.029 ms	15.093 ms	±0.053	x331
rfctest3.z	0.452 ms	0.481 ms	±0.037	x1000
randtest3.z	0.065 ms	0.072 ms	±0.011	x1000
paper-100k.pdf.z	2.208 ms	2.221 ms	±0.009	x1000
geo.protodata.z	0.897 ms	0.918 ms	±0.016	x1000

Testing

nimble test

To prevent Zippy from causing a crash or otherwise misbehaving on bad input data, a fuzzer has been run against it. You can do run the fuzzer any time by running nim c -r tests/fuzz.nim

API: zippy

import zippy

const NoCompression

NoCompression = 0

const BestSpeed

BestSpeed = 1

const BestCompression

BestCompression = 9

const DefaultCompression

DefaultCompression = -1

const HuffmanOnly

HuffmanOnly = -2

type CompressedDataFormat

Supported compressed data formats

CompressedDataFormat = enum
 dfDetect, dfZlib, dfGzip, dfDeflate

func compress

Compresses src and returns the compressed data.

func compress(src: seq[uint8]; level = DefaultCompression; dataFormat = dfGzip): seq[
 uint8] {.raises: [ZippyError].}

template compress

Helper for when preferring to work with strings.

template compress(src: string; level = DefaultCompression; dataFormat = dfGzip): string

func uncompress

Uncompresses src and returns the uncompressed data seq.

func uncompress(src: seq[uint8]; dataFormat = dfDetect): seq[uint8] {.raises: [ZippyError].}

template uncompress

Helper for when preferring to work with strings.

template uncompress(src: string; dataFormat = dfDetect): string

type ZippyError

Raised if an operation fails.

ZippyError = object of ValueError

API: zippy/tarballs

import zippy/tarballs

type EntryKind

EntryKind = enum
 ekNormalFile = 48, ekDirectory = 53

type TarballEntry

TarballEntry = object
 kind*: EntryKind
 contents*: string
 lastModified*: times.Time

type Tarball

Tarball = ref object
 contents*: OrderedTable[string, TarballEntry]

proc addDir

Recursively adds all of the files and directories inside dir to tarball.

proc addDir(tarball: Tarball; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}

proc open

Opens the tarball file located at path and reads its contents into tarball.contents (clears any existing tarball.contents entries). Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc open(tarball: Tarball; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}

proc writeTarball

Writes tarball.contents to a tarball file at path. Uses the path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc writeTarball(tarball: Tarball; path: string) {.raises: [ZippyError, IOError], tags: [WriteIOEffect].}

proc extractAll

Extracts the files stored in tarball to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).

proc extractAll(tarball: Tarball; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}

proc extractAll

Extracts the files in the tarball located at tarPath into the destination directory. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc extractAll(tarPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

proc createTarball

Creates a tarball containing all of the files and directories inside source and writes the tarball file to dest. Uses the dest path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.

proc createTarball(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}

API: zippy/ziparchives

import zippy/ziparchives

type EntryKind

EntryKind = enum
 ekFile, ekDirectory

type ArchiveEntry

ArchiveEntry = object
 kind*: EntryKind
 contents*: string

type ZipArchive

ZipArchive = ref object
 contents*: OrderedTable[string, ArchiveEntry]

proc addDir

Recursively adds all of the files and directories inside dir to archive.

proc addDir(archive: ZipArchive; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}

proc open

Opens the zip archive file located at path and reads its contents into archive.contents (clears any existing archive.contents entries).

proc open(archive: ZipArchive; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}

proc writeZipArchive

Writes archive.contents to a zip file at path.

proc writeZipArchive(archive: ZipArchive; path: string) {.raises: [ZippyError, ZippyError, IOError], tags: [WriteIOEffect].}

proc extractAll

Extracts the files stored in archive to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).

proc extractAll(archive: ZipArchive; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}

proc extractAll

Extracts the files in the archive located at zipPath into the destination directory.

proc extractAll(zipPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
 ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}

proc createZipArchive

Creates an archive containing all of the files and directories inside source and writes the zip file to dest.

proc createZipArchive(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}