zhexinli / q-vit-deit

DeiT implementation for Q-ViT

Q-ViT-DeiT

DeiT implementation for Q-ViT.

This code is built upon DeiT[https://github.com/facebookresearch/deit] and hustzxd's implementation[https://github.com/hustzxd/LSQuantization] for the LSQ paper.

Environments

PyTorch 1.7.0+ and torchvision 0.8.1+ and pytorch-image-models 0.3.2

conda install -c pytorch pytorch torchvision
pip install timm==0.3.2

How To Use

Train a float baseline using the script float_train.sh

For example:

python -m torch.distributed.launch \
--nproc_per_node=8 --use_env main.py \
--model deit_tiny_patch16_224_float \
--batch-size 256 \
--dist-eval \
--epochs 300 \
--output_dir path/to/float

Using unifrom_train.sh to train uniform-quantized qat model

For example:

wbits=4
abits=4
lr=5e-4
epochs=300
id=4bit_uniform

python -m torch.distributed.launch \
--nproc_per_node=8 --use_env main.py \
--model deit_tiny_patch16_224_mix \
--batch-size 64 \
--lr ${lr} \
--min-lr 0 \
--epochs ${epochs} \
--warmup-epochs 0 \
--wbits ${wbits} \
--abits ${abits} \
--dist-eval \
--output_dir path/to/uniform \
--finetune path/to/float

Using mixed_train.sh to train Q-ViT

For example:

wbits=5
abits=5
lr=2e-04
wd=0.05
epochs=300
lbd=1e-1
budget=21.455
id=4bit_mixed

python -m torch.distributed.launch \
--nproc_per_node=8 --use_env main.py \
--model deit_tiny_patch16_224_mix \
--batch-size 64 \
--lr ${lr} \
--min-lr 0 \
--epochs ${epochs} \
--weight-decay ${wd} \
--warmup-epochs 0 \
--wbits ${wbits} \
--abits ${abits} \
--bitops-scaler ${lbd} \
--budget ${budget} \
--stage-ratio 0.9 \
--dist-eval \
--mixpre \
--head-wise \
--output_dir path/to/mixed \
--finetune path/to/float

Here in Q-ViT, the arguments wbits and abits determines the initial bit-widths for weights and activations.

Desired Results

q-vit-deit's Issues

About Training Time

Hello, I want to ask what the training time of mixed-model is?

Size of the model

Does the size of the model reduces after this implementation . I can't see any reduction after applying this . The checkpoint size remains the same . Also inference time is not reduced and even it is worse than the float model .

Am I doing anything wrong ? or is the code provided here is wrong ?

Could anyone please confirm this ?

量化效果验证的一致性

请问量化后，分类准确率衰减与其它下游任务如目标检测、语义分割等，效果会基本一致吗？或者作者您对目前大多数的量化工作，只在分类上验证效果，有什么见解吗？

Question about the learning rate

Thanks for the great work! I have a question regarding the learning rate of uniform quantization. Does the uniform quantization use the same learning rate (i.e., 2e-4) as the mixed-precision quantization?