compressing files containing multiple similar portions about zstd

[felixhandte]

Hi @yxsamliu,

As you noted, the limiting factor here is likely the window size, which controls the maximum distance over which objects can be deduplicated. For level 6 it's 2 MB. For level 20 it's 32 MB. You can control this independently of the compression level, by setting the ZSTD_c_windowLog compression parameter, although lower compression levels have increasingly limited abilities to take advantage of very long windows.

Zstd also has a search engine specifically targeting large matches at long distances: the long-distance matcher.

For your use case, I'd expect a moderate compression level with the long-distance matcher enabled on a large window size to perform well at significantly better speed than setting a very high compression level.

[Cyan4973]

ZSTD_compressCCtx() is ignoring advanced parameters :

 *  Important : in order to behave similarly to `ZSTD_compress()`,
 *  this function compresses at requested compression level,
 *  __ignoring any other parameter__ .

Use ZSTD_compress2() instead :

 *   "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` !
 *   __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .

[Cyan4973]

If we assume that some relatively large portions (dozens of bytes at least) are very similar at byte level, but separated by significant distances, then indeed the long distance matcher would be my recommendation too. It's great at finding long matches at large distances, and the overall speed cost is reasonable. You wouldn't have to change the compression level.

Using the long distance matcher will also impact the window size automatically.
Be aware though that this mode automatically sets the window size to 128 MB by default, which might be too high. You can change that by being explicit about the window size you want.

[yxsamliu]

Thanks for your advice. I tried the following code to specify both compression level and enable LDM:

void zstd::compress(ArrayRef<uint8_t> Input,
                    SmallVectorImpl<uint8_t> &CompressedBuffer, int Level, bool EnableLdm) {
  ZSTD_CCtx* Cctx = ZSTD_createCCtx();
  if (!Cctx) report_bad_alloc_error("Failed to create ZSTD_CCtx");

  size_t const LevelStatus = ZSTD_CCtx_setParameter(Cctx, ZSTD_c_compressionLevel, Level);
  if (ZSTD_isError(LevelStatus)) {
    ZSTD_freeCCtx(Cctx);
    report_bad_alloc_error("Failed to set ZSTD_c_compressionLevel");
  }

  size_t const LdmStatus = ZSTD_CCtx_setParameter(Cctx, ZSTD_c_enableLongDistanceMatching, EnableLdm ? 1 : 0);
  printf("LDM=%d\n", (int)EnableLdm);
  if (ZSTD_isError(LdmStatus)) {
    ZSTD_freeCCtx(Cctx);
    report_bad_alloc_error("Failed to set ZSTD_c_enableLongDistanceMatching");
  }

  unsigned long CompressedBufferSize = ZSTD_compressBound(Input.size());
  CompressedBuffer.resize_for_overwrite(CompressedBufferSize);

  size_t const CompressedSize = ZSTD_compressCCtx(
      Cctx,
      CompressedBuffer.data(), CompressedBufferSize,
      Input.data(), Input.size(),
      Level
  );

  ZSTD_freeCCtx(Cctx);

  if (ZSTD_isError(CompressedSize))
    report_bad_alloc_error("Compression failed");

  __msan_unpoison(CompressedBuffer.data(), CompressedSize);
  if (CompressedSize < CompressedBuffer.size())
    CompressedBuffer.truncate(CompressedSize);
}

The compression level seems to take effect but LDM seems not. Strangely, using the zstd program with the same compression level and --long, I can see --long takes effect. Does my code missing some other parameters which makes LDM not taking effect? Thanks.

[yxsamliu]

ZSTD_compressCCtx() is ignoring advanced parameters :

 *  Important : in order to behave similarly to `ZSTD_compress()`,
 *  this function compresses at requested compression level,
 *  __ignoring any other parameter__ .

Use ZSTD_compress2() instead :

 *   "sticky" parameters are applicable to `ZSTD_compress2()` and `ZSTD_compressStream*()` !
 *   __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .

Thanks. It works.