[quantization] Add symmetric quantization schema support to Glow about glow

[qcolombet]

Since, #1444 landed, do we have anything else left here?

[rdzhabarov]

Do you think it should be more complicated than just an enum today?

Enum for symmetric, asymmetric seems to be fine.

[qcolombet]

Send a PR for the plumbing to the loader #1324.

[rdzhabarov]

The idea behind quantized sigmoid node is that CPU/Interpreter implementations are based on the prebuilt table which maps quantized input to quantized output (same for quantized tanh). Table contains a mapping (input -> output) for the input with the specific floating point range, that's why we need to select {S,O} to make sure input is within that range.

So if the backend does not want to use that mapping mechanism it will reject lowering of the quantized sigmoid/tanh. For the default case, CPU/Backend we'll make sure to restrict inputs of those nodes to some floating point range, that's why we need to find out what {S,O} correspond to that floating point range. I think it does not matter which schema will be used in the lowering of those nodes.

[rdzhabarov]

Indeed, UInt8 with a zero offset is equal to Int8 with an offset of 128.

I assume int8 with -128 offset.
Having int8 with values [-128, 127] it would require to add +128 to shift the interval to [0, 255] (given that +128 is calculated on int32 type). Which would result in offset to be reduced by 128 (given fp = x * ( q - offset)).

Now, for the motivating use case, UInt8 with offset 0, maybe we could get away with adding a new schema "symmetric with uint8 range"

From the software layer this is OK, but how would it translate to the hardware which does not support non-0 offset?

[qcolombet]

I assume int8 with -128 offset.

Exactly! I keep forgetting that the offset is subtracted not added :).

From the software layer this is OK, but how would it translate to the hardware which does not support non-0 offset

I was expecting this to be sorted out during lowering by each backend. Given that would be a specific schema, I would assume that the related support is in place before someone request it.

Admittedly I haven’t sorted out the details, I wanted to check first if it was somewhat reasonable. It sounds like it is so let us think what it practically means.

[qcolombet]

For starter, I was planning to introduce some code conditionally executed (via command line option) to force the offset to be zero.

That way we can start playing with it and see how it impacts the accuracy.

@rdzhabarov How does that sound?

[rdzhabarov]

@qcolombet I'm fine as long as it could be easily testable.

Another way would be to introduce quantizationSchema (or something like that) and pass it to the chooseQuantizationParams method. The schema itself can be controlled through the loader command line option.

[qcolombet]

Another way would be to introduce quantizationSchema (or something like that) and pass it to the chooseQuantizationParams method.

@rdzhabarov Works for me.

Do you think it should be more complicated than just an enum today?
In particular, do we want to support the cartesian product of (symmetric, asymmetric) x (zero offset, any offset)?

[qcolombet]

I was looking at the uses of chooseQuantizationParams and some of them happens during our lowering process. Most of them request symmetric ranges, so that's technically not a big deal if we don't provide a schema there, but one ask for an asymmetric range: lowerQuantizedSigmoidNode.

@rdzhabarov Any ideas on how should we tell those nodes which schema to use?

It seems weird to me to have to surface the quantization schema in the lowering API.

[qcolombet]

Introduce the symmetric schema and deal with the int8 activations (accuracy loss)

We're done with that step thanks to #1324.

[qcolombet]

For the second step, "Introduce int8, uint8 activations", @nadavrot pointed out that we could use Int8 as a canonical representation. Indeed, UInt8 with a zero offset is equal to Int8 with an offset of 128. More generally UInt8 with whatever offset O is equal to Int8 with offset O + 128. Therefore using only Int8, we simply fail to capture a small portion of what UInt8 would cover when O + 128 overflows and in my opinion, that's not worth adding a new UInt8Q type.

Now, for the motivating use case, UInt8 with offset 0, maybe we could get away with adding a new schema "symmetric with uint8 range", that would allow both offset == 0 (regular symmetric int8) and offset == 128 ("symmetric" uint8).

@rdzhabarov what do you think?

[rdzhabarov]

uint8 + symmetric schema support: #1444

[rdzhabarov]

Looks like we covered necessary parts! Thanks for working on this!