[Good First Issue][NNCF]: check number of u8, u4 constants in weight compression tests about nncf

[ljaljushkin]

Is there any link from where I can download the ImageNet dataset already formatted for testing?

Hello, @DaniAffCH! I believe you don't need such a big dataset in this context.
The task is mostly about weight compression (weight only compression) tests that validate TinyLLama model on ~20 prompts only.
Potentially, you can affect post-training quantization (or PTQ: weight + activation quantization) tests.
It that case, you can check them without ImageNet. For example, run tests that download quite small sst2 dataset:
test_ptq_quantization[hf/hf-internal-testing/tiny-random-GPTNeoXForCausalLM_backend_OPTIMUM] which uses sst2
or
test_quantize_conformance.py::test_ptq_quantization[hf/bert-base-uncased_backend_OV]

[YutingGao7]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[YutingGao7]

Hi @ljaljushkin,
I posted #2571 for tracking and reporting u8 and u4.

For item 3 "Add a check that actual number of u4 and u8 ops matches the reference values":
Is the idea to add the check to prevent regression (For example, u4 and u8 ops should not increase overtime)?
I don't see reference values defined for now. Should I use the current counts as the reference values?

[ljaljushkin]

Hi @ljaljushkin, I posted #2571 for tracking and reporting u8 and u4.

For item 3 "Add a check that actual number of u4 and u8 ops matches the reference values": Is the idea to add the check to prevent regression (For example, u4 and u8 ops should not increase overtime)? I don't see reference values defined for now. Should I use the current counts as the reference values?

Greetings @YutingGao7!
Thanks for the PR!
Yes, you are correct. Number of u4 and u8 shouldn't be changed overtime. Feel free to use the current counts as references. I will double-check them manually.

[ljaljushkin]

The assignee was unassigned due to the lack of activity.

[DaniAffCH]

.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[DaniAffCH]

Is there any link from where I can download the ImageNet dataset already formatted for testing?

[DaniAffCH]

Thank you for the clarification, I opened a PR for this!