Investigate benefits of adding BatchNorm folding to NNCF quantization about nncf

[RikAllen]

This would I think complement the FPGA implementation very well - allowing us to remove some hardware blocks.

[ljaljushkin]

Probably, it makes sense to consider similar insight from HAWQ-v3 paper:
"We follow a simpler approach where we first keep the Conv and BN layer unfolded, and then we perform standard QAT by backpropagating the gradients as usual. After several epochs, we freeze the running statistics in the BN layer and only then perform CONV+BN folding and quantized to the target bit-width"

[vshampor]

@asenina please post current results.

[asenina]

Model	Pytorch measured	Pytorch reference FP32	Openvino AC measured
unet_camvid_int8 (with folding conv and bn)	71. 57 %	71.95 %	71.67 %

[Zehaos]

@asenina Thanks you for your work. I have several questions.

1. What is "pytorch measured", "Openvino AC measured" means?

2. It seems that they got similar result on segmentation task. Is there a experiment on detection task?

If it is possible, I would be glad to test on more tasks using NNCF. Can the code (qat with folding conv and bn) be accessable to me?

[asenina]

@Zehaos

1. "pytorch measured" - accuracy was measured on final compressed model in NNCF. (Use option --mode test and --resume path-to-result-checkpoint)
   "Openvino AC measured" - Final compressed model was converted to onnx, then to IR format. Then accuracy was measured on the model in IR format (use openvino runtime and accuraсy checker)
2. I have not done experiments on detection task.

Please, track this in #446 . I will be glad if you test it on other task using NNCF and share the results (if possible). Thanks

[Zehaos]

@asenina Thanks for your reply! I will go into it soon.

[asenina]

Folding was implemented according to the following formulas:

Initial implementation
Results:

Model name	Acc (without folding)	Acc (with folding)
squeezenet1_1_imagenet_int8	58.04 %	58.14 %
unet_camvid_int8	71.66 %	72.03 %
icnet_camvid_int8	67.78 %	67.45 %
mobilenet_v2_imagnet_int8	71.31 %	71.67 %
mobilenet_v2_imagnet_int4_int8	71.03 %	70.98 %

BatchNorm folding gives minor improvements of accuracy only for int8 models. It will not merge to NNCF.