nimble install zippy
Zippy is an implementation of DEFLATE, ZLIB and GZIP data compression formats. Zippy can also create and open Tarballs (.tar, .tar.gz, .tgz, .taz) and ZIP archives.
The goal of this library is to be a pure Nim implementation that is small, performant and dependency-free.
Zippy can also be used at compile time. This is great for baking assets into executables in compressed form. Check out an example here.
To ensure Zippy is compatible with other implementations, tests/validate.nim
can be run. This script verifies that data compressed by Zippy can be uncompressed by other implementations (and that other implementations can uncompress data compressed by Zippy).
This library works well using Nim's relatively new --gc:arc
and --gc:orc
as well as the default garbage collector. This library also works using both nim c
and nim cpp
, in addition to --cc:vcc
on Windows.
I have also verified that Zippy builds with --experimental:strictFuncs
on Nim 1.4.0.
Simple examples using Zippy can be found in the examples/ folder.
- HTTP client gzip
- HTTP server gzip
- Compress a dir into a tarball or zip archive
- Extract from a tarball or zip archive
- Compose a tarball or zip archive in code
Benchmarks can be run comparing different deflate implementations. My benchmarking shows this library performs very well, a bit faster than zlib in some cases and a bit slower in others. Check the performance yourself by running tests/benchmark.nim.
nim c -d:release -r .\tests\benchmark.nim
The times below are measured on a Ryzen 5 5600X.
Each file is compressed 10 times per run.
https://github.com/guzba/zippy results:
file | min | avg | std dv | runs | % red |
---|---|---|---|---|---|
alice29.txt | 29.543 ms | 30.698 ms | ±0.188 | x167 | 63.32% |
urls.10K | 157.261 ms | 166.513 ms | ±0.449 | x31 | 67.49% |
rfctest3.gold | 5.486 ms | 5.621 ms | ±0.079 | x973 | 70.73% |
randtest3.gold | 0.757 ms | 0.777 ms | ±0.015 | x1000 | 0% |
paper-100k.pdf | 15.460 ms | 16.046 ms | ±0.109 | x331 | 19.94% |
geo.protodata | 10.012 ms | 10.434 ms | ±0.068 | x522 | 86.91% |
https://github.com/nim-lang/zip results: (Requires zlib1.dll)
file | min | avg | std dv | runs | % red |
---|---|---|---|---|---|
alice29.txt | 58.013 ms | 58.213 ms | ±0.177 | x86 | 64.23% |
urls.10K | 132.910 ms | 133.379 ms | ±0.380 | x38 | 68.29% |
rfctest3.gold | 6.525 ms | 6.589 ms | ±0.060 | x757 | 71.74% |
randtest3.gold | 0.887 ms | 0.903 ms | ±0.015 | x1000 | 0% |
paper-100k.pdf | 15.020 ms | 15.113 ms | ±0.058 | x331 | 20.59% |
geo.protodata | 8.996 ms | 9.094 ms | ±0.065 | x549 | 87.24% |
https://github.com/guzba/zippy results:
file | min | avg | std dv | runs | % red |
---|---|---|---|---|---|
alice29.txt | 12.745 ms | 13.261 ms | ±0.153 | x405 | 55.32% |
urls.10K | 40.832 ms | 41.720 ms | ±0.131 | x127 | 61.70% |
rfctest3.gold | 3.430 ms | 3.522 ms | ±0.022 | x1000 | 66.31% |
randtest3.gold | 0.291 ms | 0.306 ms | ±0.009 | x1000 | 0% |
paper-100k.pdf | 9.154 ms | 9.399 ms | ±0.049 | x600 | 18.44% |
geo.protodata | 6.609 ms | 6.747 ms | ±0.016 | x844 | 80.42% |
https://github.com/nim-lang/zip results: (Requires zlib1.dll)
file | min | avg | std dv | runs | % red |
---|---|---|---|---|---|
alice29.txt | 12.797 ms | 13.024 ms | ±0.198 | x384 | 57.17% |
urls.10K | 53.304 ms | 53.515 ms | ±0.122 | x94 | 63.93% |
rfctest3.gold | 2.179 ms | 2.220 ms | ±0.029 | x1000 | 67.53% |
randtest3.gold | 0.813 ms | 0.831 ms | ±0.016 | x1000 | 0% |
paper-100k.pdf | 12.806 ms | 12.878 ms | ±0.056 | x388 | 20.22% |
geo.protodata | 3.494 ms | 3.531 ms | ±0.024 | x1000 | 84.12% |
https://github.com/guzba/zippy results:
file | min | avg | std dv | runs | % red |
---|---|---|---|---|---|
alice29.txt | 36.293 ms | 37.635 ms | ±0.149 | x136 | 63.75% |
urls.10K | 229.110 ms | 244.351 ms | ±0.431 | x22 | 68.14% |
rfctest3.gold | 10.681 ms | 11.173 ms | ±0.121 | x465 | 70.92% |
randtest3.gold | 0.762 ms | 0.780 ms | ±0.008 | x1000 | 0% |
paper-100k.pdf | 16.124 ms | 16.833 ms | ±0.078 | x311 | 20.07% |
geo.protodata | 10.795 ms | 11.222 ms | ±0.046 | x459 | 87.07% |
https://github.com/nim-lang/zip results: (Requires zlib1.dll)
file | min | avg | std dv | runs | % red |
---|---|---|---|---|---|
alice29.txt | 86.051 ms | 86.287 ms | ±0.216 | x58 | 64.38% |
urls.10K | 253.127 ms | 253.952 ms | ±0.724 | x20 | 68.82% |
rfctest3.gold | 21.753 ms | 22.018 ms | ±0.168 | x227 | 71.77% |
randtest3.gold | 0.887 ms | 0.902 ms | ±0.014 | x1000 | 0% |
paper-100k.pdf | 16.749 ms | 16.838 ms | ±0.056 | x297 | 20.64% |
geo.protodata | 12.252 ms | 12.346 ms | ±0.066 | x405 | 87.37% |
Each file is uncompressed 10 times per run.
https://github.com/guzba/zippy results:
file | min | avg | std dv | runs |
---|---|---|---|---|
alice29.txt.z | 3.570 ms | 3.595 ms | ±0.018 | x1000 |
urls.10K.z | 17.324 ms | 17.392 ms | ±0.057 | x288 |
rfctest3.z | 0.777 ms | 0.794 ms | ±0.011 | x1000 |
randtest3.z | 0.067 ms | 0.070 ms | ±0.006 | x1000 |
paper-100k.pdf.z | 2.908 ms | 2.937 ms | ±0.034 | x1000 |
geo.protodata.z | 1.268 ms | 1.299 ms | ±0.016 | x1000 |
https://github.com/nim-lang/zip results: (Requires zlib1.dll)
file | min | avg | std dv | runs |
---|---|---|---|---|
alice29.txt.z | 3.530 ms | 3.572 ms | ±0.016 | x1000 |
urls.10K.z | 15.029 ms | 15.093 ms | ±0.053 | x331 |
rfctest3.z | 0.452 ms | 0.481 ms | ±0.037 | x1000 |
randtest3.z | 0.065 ms | 0.072 ms | ±0.011 | x1000 |
paper-100k.pdf.z | 2.208 ms | 2.221 ms | ±0.009 | x1000 |
geo.protodata.z | 0.897 ms | 0.918 ms | ±0.016 | x1000 |
nimble test
To prevent Zippy from causing a crash or otherwise misbehaving on bad input data, a fuzzer has been run against it. You can do run the fuzzer any time by running nim c -r tests/fuzz.nim
import zippy
NoCompression = 0
BestSpeed = 1
BestCompression = 9
DefaultCompression = -1
HuffmanOnly = -2
Supported compressed data formats
CompressedDataFormat = enum
dfDetect, dfZlib, dfGzip, dfDeflate
Compresses src and returns the compressed data.
func compress(src: seq[uint8]; level = DefaultCompression; dataFormat = dfGzip): seq[
uint8] {.raises: [ZippyError].}
Helper for when preferring to work with strings.
template compress(src: string; level = DefaultCompression; dataFormat = dfGzip): string
Uncompresses src and returns the uncompressed data seq.
func uncompress(src: seq[uint8]; dataFormat = dfDetect): seq[uint8] {.raises: [ZippyError].}
Helper for when preferring to work with strings.
template uncompress(src: string; dataFormat = dfDetect): string
Raised if an operation fails.
ZippyError = object of ValueError
import zippy/tarballs
EntryKind = enum
ekNormalFile = 48, ekDirectory = 53
TarballEntry = object
kind*: EntryKind
contents*: string
lastModified*: times.Time
Tarball = ref object
contents*: OrderedTable[string, TarballEntry]
Recursively adds all of the files and directories inside dir to tarball.
proc addDir(tarball: Tarball; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}
Opens the tarball file located at path and reads its contents into tarball.contents (clears any existing tarball.contents entries). Supports .tar, .tar.gz, .taz and .tgz file extensions.
proc open(tarball: Tarball; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}
Writes tarball.contents to a tarball file at path. Uses the path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.
proc writeTarball(tarball: Tarball; path: string) {.raises: [ZippyError, IOError], tags: [WriteIOEffect].}
Extracts the files stored in tarball to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).
proc extractAll(tarball: Tarball; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}
Extracts the files in the tarball located at tarPath into the destination directory. Supports .tar, .tar.gz, .taz and .tgz file extensions.
proc extractAll(tarPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}
Creates a tarball containing all of the files and directories inside source and writes the tarball file to dest. Uses the dest path's file extension to determine the tarball format. Supports .tar, .tar.gz, .taz and .tgz file extensions.
proc createTarball(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}
import zippy/ziparchives
EntryKind = enum
ekFile, ekDirectory
ArchiveEntry = object
kind*: EntryKind
contents*: string
ZipArchive = ref object
contents*: OrderedTable[string, ArchiveEntry]
Recursively adds all of the files and directories inside dir to archive.
proc addDir(archive: ZipArchive; dir: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect].}
Opens the zip archive file located at path and reads its contents into archive.contents (clears any existing archive.contents entries).
proc open(archive: ZipArchive; path: string) {.raises: [IOError, ZippyError, ZippyError], tags: [ReadIOEffect].}
Writes archive.contents to a zip file at path.
proc writeZipArchive(archive: ZipArchive; path: string) {.raises: [ZippyError, ZippyError, IOError], tags: [WriteIOEffect].}
Extracts the files stored in archive to the destination directory. The path to the destination directory must exist. The destination directory itself must not exist (it is not overwitten).
proc extractAll(archive: ZipArchive; dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadEnvEffect, ReadIOEffect, WriteDirEffect, WriteIOEffect].}
Extracts the files in the archive located at zipPath into the destination directory.
proc extractAll(zipPath, dest: string) {.raises: [IOError, ZippyError, OSError], tags: [
ReadIOEffect, ReadDirEffect, ReadEnvEffect, WriteDirEffect, WriteIOEffect].}
Creates an archive containing all of the files and directories inside source and writes the zip file to dest.
proc createZipArchive(source, dest: string) {.raises: [ZippyError, OSError, IOError], tags: [ReadDirEffect, ReadIOEffect, WriteIOEffect].}