Comments (23)
Well, then it means you are doing something wrong in the evaluation step (last step) for GZSD. Check if the commands you run for step 6 are okay.
from zsd-sc-resolver.
I got the right results!
@sandipan211, thank you so much!
from zsd-sc-resolver.
It's the code in test_mixins.py.
I changed 21 to 81 for coco, then it worked.
from zsd-sc-resolver.
It's the code in test_mixins.py.
I changed 21 to 81 for coco, then it worked.Hi @EfaJow , can you please share the path where the test_mixins.py is located? I couldnot see the test_mixins.py file inside mmdetection/mmdet.
I will also try to change (if there is 21 that needs to be replaced) ?
Thank you for your time and consideration.
from zsd-sc-resolver.
Hi,
I have provided the links to my log files for both ZSD and GZSD in my updated README.md. You can see my results there.
from zsd-sc-resolver.
Thank you so much for replying me.
I looked up the logs. They are the results after full training.
The result I posted is the result of training with only seen data.
Still appreciate your respond, I'll keep finding out what cause the zero dets of toothbrush.
from zsd-sc-resolver.
Hi, @EfaJow @sandipan211
In the command,
cd ./mmdetection
python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes seen --load_from ./work_dirs/coco2014/epoch_12.pth --save_dir ./data --data_split train
It only train on seen data. Also, following command can be used to generate test features from unseen data.
cd ./mmdetection
python tools/zero_shot_utils.py configs/faster_rcnn_r101_fpn_1x.py --classes unseen --load_from ./work_dirs/coco2014/epoch_12.pth --save_dir ./data --data_split test
Then,
python train_regressor.py
./script/train_coco_generator_65_15.sh
cd mmdetection
#evaluation on zsd
./tools/dist_test.sh configs/faster_rcnn_r101_fpn_1x.py ./work_dirs/coco2014/epoch_12.pth 1 --dataset coco --out /workspace/arijit_ug/sushil/zsd/checkpoints/ab_st_final/coco_65_15_wgan_modeSeek_seen_cycSeenUnseen_tripletSeenUnseen_varMargin_try6/coco_65_15_wgan_modeSeek_seen_cycSeenUnseen_tripletSeenUnseen_varMargin_try6_zsd_result.pkl --zsd --syn_weights /workspace/arijit_ug/sushil/zsd/checkpoints/ab_st_final/coco_65_15_wgan_modeSeek_seen_cycSeenUnseen_tripletSeenUnseen_varMargin_try6/classifier_best_latest.pth
However, it is also showing zero detections for unseen classes:
num_classes ----------- 80
+----------------+-------+-------+--------+-----------+----------+
| class | gts | dets | recall | precision | ap |
+----------------+-------+-------+--------+-----------+----------+
| person | 15697 | 48206 | 0.682 | 0.222 | 0.568102 |
| bicycle | 290 | 2940 | 0.421 | 0.041 | 0.263589 |
| car | 2392 | 20437 | 0.493 | 0.058 | 0.191976 |
| motorcycle | 118 | 1778 | 0.644 | 0.043 | 0.477236 |
| bus | 208 | 4157 | 0.659 | 0.033 | 0.181382 |
| truck | 779 | 6768 | 0.452 | 0.052 | 0.069188 |
| boat | 211 | 8705 | 0.488 | 0.012 | 0.148573 |
| traffic_light | 563 | 7972 | 0.520 | 0.037 | 0.198458 |
| fire_hydrant | 52 | 1232 | 0.808 | 0.034 | 0.612569 |
| stop_sign | 47 | 1952 | 0.766 | 0.018 | 0.487958 |
| bench | 569 | 16707 | 0.337 | 0.011 | 0.032393 |
| bird | 235 | 9234 | 0.489 | 0.012 | 0.127170 |
| dog | 437 | 2948 | 0.753 | 0.112 | 0.160115 |
| horse | 35 | 704 | 0.657 | 0.033 | 0.429912 |
| sheep | 29 | 2945 | 0.276 | 0.003 | 0.057830 |
| cow | 37 | 1507 | 0.514 | 0.013 | 0.117599 |
| elephant | 6 | 1218 | 0.333 | 0.002 | 0.190476 |
| giraffe | 7 | 729 | 0.714 | 0.007 | 0.469535 |
| backpack | 955 | 5899 | 0.440 | 0.071 | 0.085580 |
| umbrella | 320 | 5618 | 0.434 | 0.025 | 0.196311 |
| handbag | 1112 | 8904 | 0.355 | 0.044 | 0.082109 |
| tie | 160 | 2528 | 0.419 | 0.027 | 0.177361 |
| skis | 468 | 5622 | 0.323 | 0.027 | 0.072071 |
| sports_ball | 77 | 3166 | 0.545 | 0.013 | 0.066123 |
| kite | 31 | 4648 | 0.742 | 0.005 | 0.352521 |
| baseball_bat | 7 | 2391 | 0.571 | 0.002 | 0.335498 |
| baseball_glove | 3 | 1859 | 0.000 | 0.000 | 0.000000 |
| skateboard | 24 | 3027 | 0.792 | 0.006 | 0.596807 |
| surfboard | 14 | 5319 | 0.429 | 0.001 | 0.022254 |
| tennis_racket | 11 | 1340 | 0.455 | 0.004 | 0.253247 |
| bottle | 2375 | 11072 | 0.654 | 0.140 | 0.433279 |
| wine_glass | 820 | 2722 | 0.617 | 0.186 | 0.484255 |
| cup | 2730 | 12984 | 0.690 | 0.145 | 0.434292 |
| knife | 1180 | 8234 | 0.294 | 0.042 | 0.056103 |
| spoon | 999 | 7471 | 0.265 | 0.035 | 0.055141 |
| bowl | 1643 | 10518 | 0.737 | 0.115 | 0.413740 |
| banana | 183 | 3849 | 0.366 | 0.017 | 0.144224 |
| apple | 168 | 2662 | 0.327 | 0.021 | 0.084894 |
| orange | 171 | 1658 | 0.503 | 0.052 | 0.193907 |
| broccoli | 446 | 3652 | 0.419 | 0.051 | 0.158352 |
| carrot | 727 | 5285 | 0.377 | 0.052 | 0.071398 |
| pizza | 599 | 3233 | 0.656 | 0.122 | 0.399578 |
| donut | 197 | 4276 | 0.477 | 0.022 | 0.138795 |
| cake | 590 | 4936 | 0.553 | 0.066 | 0.260197 |
| chair | 2788 | 24060 | 0.462 | 0.054 | 0.196040 |
| couch | 340 | 3552 | 0.482 | 0.046 | 0.252842 |
| potted_plant | 609 | 8412 | 0.563 | 0.041 | 0.280969 |
| bed | 311 | 3359 | 0.550 | 0.051 | 0.290843 |
| dining_table | 1849 | 16013 | 0.569 | 0.066 | 0.364040 |
| tv | 934 | 5834 | 0.716 | 0.115 | 0.501795 |
| laptop | 775 | 2287 | 0.734 | 0.249 | 0.644588 |
| remote | 237 | 5218 | 0.553 | 0.025 | 0.168034 |
| keyboard | 784 | 1573 | 0.665 | 0.331 | 0.383184 |
| cell_phone | 448 | 4792 | 0.562 | 0.053 | 0.189999 |
| microwave | 83 | 1615 | 0.723 | 0.037 | 0.515616 |
| oven | 144 | 4777 | 0.514 | 0.015 | 0.255888 |
| sink | 856 | 8249 | 0.613 | 0.064 | 0.322244 |
| refrigerator | 97 | 6749 | 0.608 | 0.009 | 0.233952 |
| book | 2632 | 17148 | 0.371 | 0.057 | 0.077435 |
| clock | 172 | 5416 | 0.657 | 0.021 | 0.337422 |
| vase | 249 | 3392 | 0.538 | 0.040 | 0.194513 |
| scissors | 52 | 1752 | 0.288 | 0.009 | 0.055709 |
| teddy_bear | 130 | 1609 | 0.654 | 0.053 | 0.422264 |
| toothbrush | 137 | 2312 | 0.255 | 0.015 | 0.039345 |
+----------------+-------+-------+--------+-----------+----------+
| mean | | | 0.524 | | 0.251232 |
+----------------+-------+-------+--------+-----------+----------+
+---------------+------+------+--------+-----------+----------+
| class | gts | dets | recall | precision | ap |
+---------------+------+------+--------+-----------+----------+
| airplane | 1444 | 436 | 0.015 | 0.048 | 0.001692 |
| train | 1602 | 880 | 0.002 | 0.005 | 0.000014 |
| parking_meter | 510 | 2714 | 0.073 | 0.014 | 0.001345 |
| cat | 1669 | 0 | 0.000 | 0.000 | 0.000000 |
| bear | 462 | 152 | 0.030 | 0.092 | 0.006084 |
| suitcase | 2219 | 219 | 0.005 | 0.046 | 0.000246 |
| frisbee | 935 | 70 | 0.000 | 0.000 | 0.000000 |
| snowboard | 793 | 326 | 0.001 | 0.003 | 0.000049 |
| fork | 1775 | 0 | 0.000 | 0.000 | 0.000000 |
| sandwich | 1457 | 319 | 0.007 | 0.031 | 0.000232 |
| hot_dog | 1009 | 4315 | 0.071 | 0.017 | 0.001760 |
| toilet | 1462 | 100 | 0.007 | 0.100 | 0.000998 |
| mouse | 850 | 335 | 0.000 | 0.000 | 0.000000 |
| toaster | 78 | 32 | 0.000 | 0.000 | 0.000000 |
| hair_drier | 74 | 3459 | 0.014 | 0.000 | 0.000004 |
+---------------+------+------+--------+-----------+----------+
| mean | | | 0.015 | | 0.000828 |
+---------------+------+------+--------+-----------+----------+
+------+--+--+-------+--+----------+
| mean | | | 0.524 | | 0.251232 |
+------+--+--+-------+--+----------+
| mean | | | 0.015 | | 0.000828 |
+------+--+--+-------+--+----------+
mAP is : 0.20368660986423492
I will be grateful if you can suggest the better solution so that detections on unseen data can also be obtained. I followed all the steps given in https://github.com/sandipan211/ZSD-SC-Resolver for the training and evaluation.
Thank you for your time and consideration.
Hi,
I have provided the links to my log files for both ZSD and GZSD in my updated README.md. You can see my results there.
from zsd-sc-resolver.
Thank you so much for replying me. I looked up the logs. They are the results after full training. The result I posted is the result of training with only seen data. Still appreciate your respond, I'll keep finding out what cause the zero dets of toothbrush.
from zsd-sc-resolver.
Hi @chiran7 ,
I have provided our trained classifier weights that ultimately impact the unseen classes. You can find them in the README.md. Can you kindly evaluate with those weights? If the results are reproduced, that will mean the problem you are facing is probably in step 5. Also, can you please tell me your environment settings?
from zsd-sc-resolver.
Hi @EfaJow ,
If you had tested after training with seen data only (I believe that is till step 3), the low results you are seeing for unseen classes is expected, as zero-shot capabilities aren't learned by training with seen data only. That step (epoch_12.pth) was only training the Faster-RCNN for detecting a seen object. Our novelty starts after step 3, where unseen detection capabilities are achieved. Once you run all 5 steps and then evaluate, I think your unseen mAP will improve.
If you have run all the steps in README and still achieving low unseen mAP, kindly refer to my response to @chiran7.
Hope this helps.
from zsd-sc-resolver.
Hi @chiran7 ,
I have provided our trained classifier weights that ultimately impact the unseen classes. You can find them in the README.md. Can you kindly evaluate with those weights? If the results are reproduced, that will mean the problem you are facing is probably in step 5. Also, can you please tell me your environment settings?
Dear @sandipan211 ,
Thank you for the information. I have used the weight provided in README.md, and still it shows results for seen but zero for unseen. In step 4, the tran_regressor.py only trains on seen data, and step 5 uses weights of seen data to train GAN on unseen data. In terms of the environmental setting, I have not used distributed training for step 2 and trained only (without validation). I have installed mmdetection and mmcv packages without using anaconda environment in Linux environments to train the model.
Can you please explain in detail how to use unseen features learning by training on seen class in steps 4 and 5?
Thank you for your time and consideration.
from zsd-sc-resolver.
README.md
Dear @sandipan211 ,
Thank you for your time. It worked now. I used the trained classifier weights instead of downloading from README.md. After training all 5 steps and using the trained weight, the evaluation step is showing good results on unseen classes for the MS COCO dataset.
from zsd-sc-resolver.
Hi @sandipan211 ,
I followed all the steps and tune the hyper parameters due to the paper, however, the mAP score on unseen is only 7. I didn't change the code, but I must missed something. Could you please help me to get the right score? I really appreciate your responds.
Here is the final result, and the script I runned.
+----------------+-------+--------+--------+-----------+----------+
| class | gts | dets | recall | precision | ap |
+----------------+-------+--------+--------+-----------+----------+
| person | 15697 | 197511 | 0.895 | 0.071 | 0.660126 |
| bicycle | 290 | 4315 | 0.628 | 0.042 | 0.294972 |
| car | 2392 | 47730 | 0.788 | 0.039 | 0.192863 |
| motorcycle | 118 | 1993 | 0.720 | 0.043 | 0.510872 |
| bus | 208 | 7529 | 0.707 | 0.020 | 0.196596 |
| truck | 779 | 11096 | 0.413 | 0.029 | 0.039762 |
| boat | 211 | 17607 | 0.616 | 0.007 | 0.174057 |
| traffic_light | 563 | 18147 | 0.650 | 0.020 | 0.173153 |
| fire_hydrant | 52 | 932 | 0.827 | 0.046 | 0.546072 |
| stop_sign | 47 | 1355 | 0.745 | 0.026 | 0.521156 |
| bench | 569 | 37232 | 0.489 | 0.007 | 0.049017 |
| bird | 235 | 31394 | 0.523 | 0.004 | 0.124825 |
| dog | 437 | 8228 | 0.787 | 0.042 | 0.178870 |
| horse | 35 | 615 | 0.629 | 0.036 | 0.453032 |
| sheep | 29 | 8040 | 0.310 | 0.001 | 0.022340 |
| cow | 37 | 1301 | 0.595 | 0.017 | 0.140441 |
| elephant | 6 | 278 | 0.500 | 0.011 | 0.500000 |
| giraffe | 7 | 376 | 0.857 | 0.016 | 0.563532 |
| backpack | 955 | 8703 | 0.465 | 0.051 | 0.054644 |
| umbrella | 320 | 6652 | 0.672 | 0.032 | 0.225764 |
| handbag | 1112 | 13246 | 0.423 | 0.035 | 0.082643 |
| tie | 160 | 7759 | 0.613 | 0.013 | 0.102578 |
| skis | 468 | 20450 | 0.560 | 0.013 | 0.044583 |
| sports_ball | 77 | 12764 | 0.545 | 0.003 | 0.035616 |
| kite | 31 | 11229 | 0.742 | 0.002 | 0.248010 |
| baseball_bat | 7 | 1671 | 0.857 | 0.004 | 0.139171 |
| baseball_glove | 3 | 1527 | 0.000 | 0.000 | 0.000000 |
| skateboard | 24 | 2984 | 0.833 | 0.007 | 0.605755 |
| surfboard | 14 | 8082 | 0.357 | 0.001 | 0.011808 |
| tennis_racket | 11 | 1696 | 0.455 | 0.003 | 0.200000 |
| bottle | 2375 | 21580 | 0.742 | 0.082 | 0.392164 |
| wine_glass | 820 | 3955 | 0.685 | 0.142 | 0.467229 |
| cup | 2730 | 14137 | 0.688 | 0.133 | 0.401042 |
| knife | 1180 | 10947 | 0.397 | 0.043 | 0.052717 |
| spoon | 999 | 16724 | 0.414 | 0.025 | 0.048997 |
| bowl | 1643 | 10824 | 0.732 | 0.111 | 0.374777 |
| banana | 183 | 7671 | 0.628 | 0.015 | 0.101547 |
| apple | 168 | 2889 | 0.429 | 0.025 | 0.085480 |
| orange | 171 | 2735 | 0.702 | 0.044 | 0.123484 |
| broccoli | 446 | 18503 | 0.749 | 0.018 | 0.117124 |
| carrot | 727 | 20421 | 0.707 | 0.025 | 0.060947 |
| pizza | 599 | 5221 | 0.755 | 0.087 | 0.421445 |
| donut | 197 | 7519 | 0.594 | 0.016 | 0.143523 |
| cake | 590 | 5883 | 0.627 | 0.063 | 0.243434 |
| chair | 2788 | 69915 | 0.655 | 0.026 | 0.197117 |
| couch | 340 | 2774 | 0.494 | 0.061 | 0.210323 |
| potted_plant | 609 | 25273 | 0.798 | 0.019 | 0.261391 |
| bed | 311 | 5224 | 0.707 | 0.042 | 0.216464 |
| dining_table | 1849 | 98057 | 0.786 | 0.015 | 0.192642 |
| tv | 934 | 9977 | 0.760 | 0.071 | 0.426272 |
| laptop | 775 | 3459 | 0.796 | 0.178 | 0.631535 |
| remote | 237 | 8552 | 0.603 | 0.017 | 0.129786 |
| keyboard | 784 | 6502 | 0.777 | 0.094 | 0.297538 |
| cell_phone | 448 | 6936 | 0.592 | 0.038 | 0.163256 |
| microwave | 83 | 620 | 0.687 | 0.092 | 0.593161 |
| oven | 144 | 6156 | 0.688 | 0.016 | 0.286441 |
| sink | 856 | 32820 | 0.821 | 0.021 | 0.255408 |
| refrigerator | 97 | 15172 | 0.732 | 0.005 | 0.301248 |
| book | 2632 | 51358 | 0.697 | 0.036 | 0.083455 |
| clock | 172 | 8410 | 0.680 | 0.014 | 0.196000 |
| vase | 249 | 3571 | 0.546 | 0.038 | 0.185590 |
| scissors | 52 | 2044 | 0.404 | 0.010 | 0.115085 |
| teddy_bear | 130 | 2963 | 0.838 | 0.037 | 0.415707 |
| toothbrush | 137 | 2429 | 0.394 | 0.022 | 0.066894 |
+----------------+-------+--------+--------+-----------+----------+
| mean | | | 0.633 | | 0.239867 |
+----------------+-------+--------+--------+-----------+----------+
+---------------+------+------+--------+-----------+----------+
| class | gts | dets | recall | precision | ap |
+---------------+------+------+--------+-----------+----------+
| airplane | 1444 | 42 | 0.019 | 0.643 | 0.016029 |
| train | 1602 | 550 | 0.233 | 0.680 | 0.194064 |
| parking_meter | 510 | 1 | 0.000 | 0.000 | 0.000000 |
| cat | 1669 | 1473 | 0.533 | 0.604 | 0.390125 |
| bear | 462 | 40 | 0.078 | 0.900 | 0.073715 |
| suitcase | 2219 | 441 | 0.066 | 0.333 | 0.023340 |
| frisbee | 935 | 25 | 0.002 | 0.080 | 0.000357 |
| snowboard | 793 | 471 | 0.274 | 0.461 | 0.195434 |
| fork | 1775 | 259 | 0.064 | 0.440 | 0.037449 |
| sandwich | 1457 | 1793 | 0.365 | 0.297 | 0.138656 |
| hot_dog | 1009 | 3 | 0.000 | 0.000 | 0.000000 |
| toilet | 1462 | 512 | 0.116 | 0.332 | 0.040305 |
| mouse | 850 | 105 | 0.015 | 0.124 | 0.002790 |
| toaster | 78 | 97 | 0.026 | 0.021 | 0.002999 |
| hair_drier | 74 | 8 | 0.000 | 0.000 | 0.000000 |
+---------------+------+------+--------+-----------+----------+
| mean | | | 0.119 | | 0.074351 |
+---------------+------+------+--------+-----------+----------+
+------+--+--+-------+--+----------+
| mean | | | 0.633 | | 0.239867 |
+------+--+--+-------+--+----------+
| mean | | | 0.119 | | 0.074351 |
+------+--+--+-------+--+----------+
mAP is : 0.20843979716300964
python trainer.py --manualSeed 806
--cls_weight 0.01 --cls_weight_unseen 0 --nclass_all 81 --syn_num 250 --val_every 1
--cuda --netG_name MLP_G --netD_name MLP_D
--nepoch 55 --nepoch_cls 15 --ngh 4096 --ndh 4096 --lambda1 10 --critic_iter 5
--dataset coco --batch_size 128 --nz 300 --attSize 300 --resSize 1024 --gan_epoch_budget 50000
--lr 0.0005 --lr_step 30 --lr_cls 0.0001
--pretrain_classifier mmdetection/work_dirs/epoch_12.pth
--class_embedding MSCOCO/fasttext.npy
--dataroot mmdetection/data/features/coco/65_15_epoch12
--testsplit test_0.7_0.3
--trainsplit train_0.7_0.3
--classes_split 65_15
--lz_ratio 0.01
--outname checkpoints/resolver_base
--regressor_lamda 0.01
--triplet_lamda 0.1
--pretrain_regressor MSCOCO/regressor_1.pth --tr_mu_dtilde 0.5 --tr_sigma_dtilde 0.5 \
from zsd-sc-resolver.
I also tested the weight of classifier for coco provided in README.md, the unseen mAP is only 10. I really don't get it. Could you please help?
+----------------+-------+--------+--------+-----------+----------+
| class | gts | dets | recall | precision | ap |
+----------------+-------+--------+--------+-----------+----------+
| person | 15697 | 197379 | 0.895 | 0.071 | 0.660127 |
| bicycle | 290 | 4309 | 0.628 | 0.042 | 0.294972 |
| car | 2392 | 47654 | 0.788 | 0.040 | 0.192863 |
| motorcycle | 118 | 1994 | 0.720 | 0.043 | 0.510872 |
| bus | 208 | 7502 | 0.707 | 0.020 | 0.196596 |
| truck | 779 | 11081 | 0.413 | 0.029 | 0.039763 |
| boat | 211 | 17589 | 0.616 | 0.007 | 0.173923 |
| traffic_light | 563 | 18121 | 0.650 | 0.020 | 0.173154 |
| fire_hydrant | 52 | 927 | 0.827 | 0.046 | 0.546091 |
| stop_sign | 47 | 1353 | 0.745 | 0.026 | 0.521158 |
| bench | 569 | 37164 | 0.487 | 0.007 | 0.049004 |
| bird | 235 | 31332 | 0.523 | 0.004 | 0.124819 |
| dog | 437 | 8222 | 0.787 | 0.042 | 0.178869 |
| horse | 35 | 613 | 0.629 | 0.036 | 0.453032 |
| sheep | 29 | 8039 | 0.310 | 0.001 | 0.022340 |
| cow | 37 | 1299 | 0.595 | 0.017 | 0.140441 |
| elephant | 6 | 278 | 0.500 | 0.011 | 0.500000 |
| giraffe | 7 | 375 | 0.857 | 0.016 | 0.563532 |
| backpack | 955 | 8708 | 0.465 | 0.051 | 0.054644 |
| umbrella | 320 | 6648 | 0.672 | 0.032 | 0.225764 |
| handbag | 1112 | 13226 | 0.423 | 0.036 | 0.082654 |
| tie | 160 | 7756 | 0.613 | 0.013 | 0.102579 |
| skis | 468 | 20447 | 0.560 | 0.013 | 0.044584 |
| sports_ball | 77 | 12756 | 0.545 | 0.003 | 0.035616 |
| kite | 31 | 11197 | 0.742 | 0.002 | 0.248010 |
| baseball_bat | 7 | 1670 | 0.857 | 0.004 | 0.139171 |
| baseball_glove | 3 | 1525 | 0.000 | 0.000 | 0.000000 |
| skateboard | 24 | 2983 | 0.833 | 0.007 | 0.605755 |
| surfboard | 14 | 8069 | 0.357 | 0.001 | 0.011808 |
| tennis_racket | 11 | 1696 | 0.455 | 0.003 | 0.200000 |
| bottle | 2375 | 21571 | 0.742 | 0.082 | 0.392163 |
| wine_glass | 820 | 3944 | 0.685 | 0.143 | 0.467232 |
| cup | 2730 | 14122 | 0.688 | 0.133 | 0.401050 |
| knife | 1180 | 10870 | 0.396 | 0.043 | 0.052664 |
| spoon | 999 | 16653 | 0.412 | 0.025 | 0.048935 |
| bowl | 1643 | 10803 | 0.732 | 0.111 | 0.374773 |
| banana | 183 | 7676 | 0.628 | 0.015 | 0.101546 |
| apple | 168 | 2880 | 0.423 | 0.025 | 0.085310 |
| orange | 171 | 2735 | 0.702 | 0.044 | 0.123481 |
| broccoli | 446 | 18510 | 0.749 | 0.018 | 0.117117 |
| carrot | 727 | 20410 | 0.707 | 0.025 | 0.060948 |
| pizza | 599 | 5207 | 0.755 | 0.087 | 0.421447 |
| donut | 197 | 7523 | 0.594 | 0.016 | 0.143523 |
| cake | 590 | 5875 | 0.625 | 0.063 | 0.243328 |
| chair | 2788 | 69867 | 0.655 | 0.026 | 0.197104 |
| couch | 340 | 2770 | 0.494 | 0.061 | 0.210324 |
| potted_plant | 609 | 25250 | 0.798 | 0.019 | 0.261394 |
| bed | 311 | 5219 | 0.707 | 0.042 | 0.216468 |
| dining_table | 1849 | 97965 | 0.786 | 0.015 | 0.192640 |
| tv | 934 | 9981 | 0.760 | 0.071 | 0.426271 |
| laptop | 775 | 3455 | 0.796 | 0.179 | 0.631536 |
| remote | 237 | 8544 | 0.608 | 0.017 | 0.129873 |
| keyboard | 784 | 6498 | 0.777 | 0.094 | 0.297537 |
| cell_phone | 448 | 6937 | 0.592 | 0.038 | 0.163256 |
| microwave | 83 | 617 | 0.687 | 0.092 | 0.593165 |
| oven | 144 | 6159 | 0.688 | 0.016 | 0.286442 |
| sink | 856 | 32855 | 0.822 | 0.021 | 0.255445 |
| refrigerator | 97 | 15191 | 0.732 | 0.005 | 0.300810 |
| book | 2632 | 51343 | 0.697 | 0.036 | 0.083468 |
| clock | 172 | 8407 | 0.680 | 0.014 | 0.196001 |
| vase | 249 | 3570 | 0.546 | 0.038 | 0.185590 |
| scissors | 52 | 2040 | 0.404 | 0.010 | 0.115085 |
| teddy_bear | 130 | 2960 | 0.838 | 0.037 | 0.415708 |
| toothbrush | 137 | 2427 | 0.394 | 0.022 | 0.066894 |
+----------------+-------+--------+--------+-----------+----------+
| mean | | | 0.633 | | 0.239854 |
+----------------+-------+--------+--------+-----------+----------+
+---------------+------+------+--------+-----------+----------+
| class | gts | dets | recall | precision | ap |
+---------------+------+------+--------+-----------+----------+
| airplane | 1444 | 189 | 0.089 | 0.677 | 0.068682 |
| train | 1602 | 868 | 0.382 | 0.705 | 0.322311 |
| parking_meter | 510 | 21 | 0.002 | 0.048 | 0.000280 |
| cat | 1669 | 1681 | 0.610 | 0.606 | 0.476787 |
| bear | 462 | 161 | 0.284 | 0.814 | 0.244931 |
| suitcase | 2219 | 432 | 0.070 | 0.361 | 0.029319 |
| frisbee | 935 | 28 | 0.002 | 0.071 | 0.000481 |
| snowboard | 793 | 482 | 0.247 | 0.407 | 0.171431 |
| fork | 1775 | 1298 | 0.211 | 0.288 | 0.093899 |
| sandwich | 1457 | 1281 | 0.281 | 0.320 | 0.108900 |
| hot_dog | 1009 | 1 | 0.000 | 0.000 | 0.000000 |
| toilet | 1462 | 121 | 0.035 | 0.421 | 0.018076 |
| mouse | 850 | 52 | 0.001 | 0.019 | 0.000056 |
| toaster | 78 | 119 | 0.026 | 0.017 | 0.003257 |
| hair_drier | 74 | 6 | 0.000 | 0.000 | 0.000000 |
+---------------+------+------+--------+-----------+----------+
| mean | | | 0.149 | | 0.102561 |
+---------------+------+------+--------+-----------+----------+
+------+--+--+-------+--+----------+
| mean | | | 0.633 | | 0.239854 |
+------+--+--+-------+--+----------+
| mean | | | 0.149 | | 0.102561 |
+------+--+--+-------+--+----------+
mAP is : 0.2137858271598816
from zsd-sc-resolver.
Hi @EfaJow,
On comparing the training arguments as given by you here and mine as given here, I can see that the learning rate (lr) used by you is different. Can you retry with the learning rate I used? Kindly retry and cross-verify every step again and check using the weights I uploaded.
from zsd-sc-resolver.
Hi @sandipan211,
Thank you for your response.
The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063.
+------+--+--+-------+--+----------+
| mean | | | 0.633 | | 0.239862 |
+------+--+--+-------+--+----------+
| mean | | | 0.140 | | 0.091063 |
+------+--+--+-------+--+----------+
I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.
from zsd-sc-resolver.
Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+
I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.
It's okay, I can understand it is a complex framework and lots of things can go wrong. However, I fail to understand why this is happening. Others like @chiran7 also experienced something similar but were able to reproduce results with pretrained classifier weights as you can see here. I hope you have performed each and every step as written in readme.
Kindly tell me your environment details as well. You can refer to this issue on how I created my environment. Although it seems unlikely that performance is related to environment settings in this case, you might give it a try as well.
from zsd-sc-resolver.
Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+
I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.
Moreover, it seems like you tested in the GZSD setting. Can you run in the ZSD setting once and tell me how much mAP you're able to obtain?
from zsd-sc-resolver.
Hi @sandipan211, Thank you for your response. The learning rate I used is due to the implementation details part of your paper. I also tried to use 0.00005 as lr, but the result mAP is 0.239862 and 0.091063. +------+--+--+-------+--+----------+ | mean | | | 0.633 | | 0.239862 | +------+--+--+-------+--+----------+ | mean | | | 0.140 | | 0.091063 | +------+--+--+-------+--+----------+
I also tested the classifier weights required in the evaluation step given in the readme in the step 6, but the mAP result is only 0.239854 and 0.102561, not as high as shown in your log. Could you provide any idea about this? I don't want to look annoying, but I'm at my wits' end with this problem. Thanks for your help.Moreover, it seems like you tested in the GZSD setting. Can you run in the ZSD setting once and tell me how much mAP you're able to obtain?
I tried just now, the result is right in ZSD! It's 0.200599.
from zsd-sc-resolver.
from zsd-sc-resolver.
README.md
Dear @sandipan211 , Thank you for your time. It worked now. I used the trained classifier weights instead of downloading from README.md. After training all 5 steps and using the trained weight, the evaluation step is showing good results on unseen classes for the MS COCO dataset.
Hi @chiran7,
Did you adjust the code in test_mixins.py to get the right mAP result in GZSD on COCO?
Could you share how you get the right mAP in GZSD (which is 0.374 in seen and 0.201 in unseen on COCO)?
from zsd-sc-resolver.
Great to know that @EfaJow ! You've been on it for months, so glad to hear it worked out! I'm curious - what was the issue?
from zsd-sc-resolver.
It's the code in test_mixins.py.
I changed 21 to 81 for coco, then it worked.
Hi @EfaJow , can you please share the path where the test_mixins.py is located? I couldnot see the test_mixins.py file inside mmdetection/mmdet.
I will also try to change (if there is 21 that needs to be replaced) ?
Thank you for your time and consideration.
from zsd-sc-resolver.
Related Issues (20)
- Training on custom data HOT 2
- class embedding vector for custom data HOT 4
- Custom Data HOT 1
- low accuracy on unseen classes HOT 1
- when classifier is trained
- is train_unseen_classifier.py missing HOT 1
- how to generate test npy for unseen classes HOT 2
- custom dataset tune HOT 2
- Pascal VOC data split HOT 2
- inference to single image HOT 1
- Can you provide some output files? HOT 7
- Author, Do you think you will update the code and environment to the latest version? HOT 2
- I have some questions about training on VOC datasets HOT 5
- Loss suddenly becoming very large
- Do you have a minimal environment YAML file? HOT 1
- Do you still have the weight file obtained by executing the fifth step HOT 2
- Do you think you'll ever migrate your code to the latest stable versions of mmcv and mmdetection? HOT 13
- Some hesitations related to background class label index HOT 5
- Confusion about validation metrics for zero-shot detection. HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zsd-sc-resolver.