Configurable averageBits about chunker

[fd0]

You're welcome. At the time I wrote the package, making averageBits a constant led to much faster code, and since it isn't necessary for restic to configure this, it was left as a constant.

I would consider a PR which adds configuration options for this, if it happens in a backwards compatible way and is not significantly slower than the old code with Go >= 1.8.

What do you plan to do with the chunker?

[vitalyisaev2]

I would consider a PR which adds configuration options for this, if it happens in a backwards compatible way

I guess that the only option to extend library interface is to add an extra constructor for Chunker, something like:

func NewWithCustomAverageBits(rd io.Reader, pol Pol, int averageBits) *Chunker {
    // ...
}

Also, averageBits should become either a global scope variable, or a chunkerConfig member.

and is not significantly slower than the old code with Go >= 1.8

Do you mean that changing averageBits from constant to variable would reduce chunker performance?

What do you plan to do with the chunker?

I'm doing some research on a storage optimizations (especially on data deduplication), and I faced with a use-case where averageBits should be slightly reduced in order to achive better deduplication. A good example is a stream of small text files with little pairwise differences.

[fd0]

I guess that the only option to extend library interface is to add an extra constructor for Chunker, something like:

What about adding a method on the *Chunker, like SetAverageBits(bits int)?

Also, averageBits should become either a global scope variable, or a chunkerConfig member.

Nonono, no global variable, it must be specific to a single *Chunker. Global state is bad.

Do you mean that changing averageBits from constant to variable would reduce chunker performance?

Exactly, at least for old versions of Go (like 1.3 or so).

I'm doing some research on a storage optimizations (especially on data deduplication), and I faced with a use-case where averageBits should be slightly reduced in order to achive better deduplication. A good example is a stream of small text files with little pairwise differences.

Ah, that makes sense. Thanks for explaining!