Training Performance Degradation about nncf

[ljaljushkin]

Thanks, VS for the great reproducer!
I've root caused and fixed the issue. The reason isn't mixed-precision initialization, but quantization metrics collection on each train iteration.

            for stat_name, stat_value in compression_ctrl.statistics().items():
                if isinstance(stat_value, (int, float)):

Starting from #33 it takes a lot of time.
Here's the fix - #287

After the fix, 1 epoch takes 52 min with RTX-2080, instead of 76 min before.

[vshampor]

@vuiseng9 are you sure that torch 1.6.2 was used? We automatically install 1.5.0 by default (not reinstalling it, though, if a newer version is already present, AFAIK)

[vuiseng9]

@vshampor Sorry, it is a typo, it is 1.6.0. I notice torch version will be different if we install python setup.py install vs pip install -r requirements.txt

[kchechil]

@vuiseng9 @ljaljushkin is this still actual?